-- Logs begin at Fri 2022-07-29 03:28:39 UTC, end at Fri 2022-07-29 05:15:09 UTC. -- Jul 29 04:32:54.989907 dv-cinder-851449-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Jul 29 04:32:54.997098 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Jul 29 04:32:54.999112 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: *** Starting uWSGI 2.0.18-debian (64bit) on [Fri Jul 29 04:32:54 2022] *** Jul 29 04:32:54.999112 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Jul 29 04:32:54.999112 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Jul 29 04:32:54.999112 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: nodename: dv-cinder-851449-2-iscsi Jul 29 04:32:54.999112 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: machine: x86_64 Jul 29 04:32:54.999112 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: clock source: unix Jul 29 04:32:54.999499 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: pcre jit disabled Jul 29 04:32:54.999499 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: detected number of CPU cores: 4 Jul 29 04:32:54.999499 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: current working directory: / Jul 29 04:32:54.999499 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: detected binary path: /usr/bin/uwsgi-core Jul 29 04:32:54.999499 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: your processes number limit is 47680 Jul 29 04:32:54.999499 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: your memory page size is 4096 bytes Jul 29 04:32:54.999499 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: detected max file descriptor number: 2048 Jul 29 04:32:54.999499 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: lock engine: pthread robust mutexes Jul 29 04:32:54.999499 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: thunder lock: enabled Jul 29 04:32:54.999499 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Jul 29 04:32:55.000084 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Jul 29 04:32:55.040472 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: Python main interpreter initialized at 0x563d117e19c0 Jul 29 04:32:55.040472 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: python threads support enabled Jul 29 04:32:55.040472 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: your server socket listen backlog is limited to 100 connections Jul 29 04:32:55.040472 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: your mercy for graceful operations on workers is 90 seconds Jul 29 04:32:55.040879 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: mapped 403077 bytes (393 KB) for 2 cores Jul 29 04:32:55.040956 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: *** Operational MODE: preforking *** Jul 29 04:32:55.041220 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: *** uWSGI is running in multiple interpreter mode *** Jul 29 04:32:55.041220 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: spawned uWSGI master process (pid: 92180) Jul 29 04:32:55.041422 dv-cinder-851449-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Jul 29 04:32:55.042825 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: spawned uWSGI worker 1 (pid: 92189, cores: 1) Jul 29 04:32:55.042825 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: spawned uWSGI worker 2 (pid: 92190, cores: 1) Jul 29 04:32:55.042825 dv-cinder-851449-2-iscsi devstack@keystone.service[92180]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Jul 29 04:33:00.553562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 29 04:33:00.553562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: from cryptography.utils import int_from_bytes Jul 29 04:33:00.554341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 29 04:33:00.554341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: from cryptography.utils import int_from_bytes Jul 29 04:33:00.554454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 29 04:33:00.554454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: from cryptography.utils import int_from_bytes Jul 29 04:33:00.554859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 29 04:33:00.554859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: from cryptography.utils import int_from_bytes Jul 29 04:33:00.604882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 29 04:33:00.605086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 29 04:33:00.605260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 29 04:33:00.605352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] command line args: [] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 29 04:33:00.605455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 29 04:33:00.605533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 29 04:33:00.605648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] command line args: [] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 29 04:33:00.605725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ================================================================================ {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 29 04:33:00.605918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 29 04:33:00.605996 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] admin_token = **** {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.606096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ================================================================================ {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 29 04:33:00.606203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] config_dir = [] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.606352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] admin_token = **** {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.606435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.606572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] config_source = [] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.606647 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] config_dir = [] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.606759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] debug = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.606846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.606943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] config_source = [] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.607024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] debug = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] insecure_debug = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] insecure_debug = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608788 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] list_limit = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] log_config_append = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.608953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] list_limit = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] log_config_append = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] log_dir = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] log_dir = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] log_file = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] log_file = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.609959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.610046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.610107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.610169 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.610287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.610407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.610533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.610656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.610777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.610896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.611018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.611137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.611251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.611368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.611486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.611605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.611731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.611853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.611961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.612081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.612193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] notification_format = cadf {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.612310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] notification_format = cadf {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.612431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.612546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.612821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.40/identity {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.613021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.40/identity {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.613158 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] publish_errors = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.613279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] publish_errors = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.613413 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.613535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.613661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.613783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.613902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.614016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.614133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.614265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.614390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.614511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.614624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] standard_threads = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.614736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] standard_threads = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.614849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] strict_password_check = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.615026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] strict_password_check = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.615143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.615143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] use_eventlog = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.615352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.615352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] use_eventlog = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.615560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] use_journal = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.615681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] use_journal = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.615795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] use_json = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.615914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] use_json = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.616040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] use_stderr = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.616154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] use_stderr = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.616275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] use_syslog = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.616389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] use_syslog = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.616507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] watch_log_file = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.616651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] watch_log_file = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 29 04:33:00.616774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.616888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.617012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.617132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.617246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.617358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.617470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.617588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.617710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.617824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.617943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.618058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.618170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.618288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.618413 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.618528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.618640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.618751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.618863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.618974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.619099 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.619213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.619325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.619437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.619548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.619671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.619793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.619911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.620043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.620165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.620278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.620390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.620584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.620717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.620830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.620941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.621059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.621183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.621306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.621420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.621538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.621651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.621763 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.621880 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.622002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.622115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.622228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.622340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.622451 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.622562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.622686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.622799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.622912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.623023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.623134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.623246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.623366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.623479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.623596 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.623709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.623820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.623940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.624054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.624165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.624165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.624381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.624494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.624632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.624758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.624873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.624985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.625096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.625215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.625338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.625452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.625569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.625688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.625801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.625918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.626107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.626222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.626334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.626446 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.626556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.626674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.626795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.626909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.627020 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.627132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.627256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.627376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.627506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.627621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.627732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.627827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.627919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.628013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.628113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.628208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.628300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.628393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.628492 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.628587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.628756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.628854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.629015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.629202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.629323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.629467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.629600 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.629708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.629820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.629932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.630044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.630145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.630257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.630378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.630493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.630607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.630719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.630831 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.630945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.631066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.631187 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.631310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.631433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.631555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.631677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.631780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.631900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.632021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.632135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.632248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.632367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.632480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.632616 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.632743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.632849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.632961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.633073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.633185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.633307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.633427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.633542 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.633654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.633654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.633875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.633990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.634103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.634215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.enabled = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.634327 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.634440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.634560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.enabled = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.634673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.634787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.634900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.635013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.635132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.635253 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.635368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.635480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.635592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.635704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.635817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.635937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.636051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.636164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.636276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.636388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.636500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.636676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.636795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.636910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.637029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.637142 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.637254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.637445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.637570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.637685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.637804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.637919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.638031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.638152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.638274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.638400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.638514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.638626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.638739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.638852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.638966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.639087 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.639207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.639321 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.639434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.639550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.639663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.639787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.639902 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.640015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.640128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.640240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.640353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.640477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.640592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.640720 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.640833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.640945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.641069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] auth.external = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.641192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.641306 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] auth.mapped = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.641419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] auth.external = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.641532 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.641645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] auth.mapped = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.641787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.641910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.642035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] auth.password = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.642151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.642265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] auth.token = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.642379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] auth.password = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.642492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.642606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] auth.token = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.642728 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] catalog.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.642844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.642957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.643079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] catalog.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.643195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.643309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.643429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.643551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.643666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.643780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.643894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.643894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] credential.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.644113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.644229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] credential.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.644343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.644343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] credential.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.644562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.644562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.645318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] credential.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.645435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.645550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.645664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.645777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.645891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.646017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.646133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.646247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.646362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.646362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.646575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.646697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.646812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.646926 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.647040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.647153 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.647275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.647399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.647515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.647628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.647741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.647855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.647968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.648076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] federation.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.648174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.648268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.648370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.648467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] federation.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.648563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.648700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.648801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] federation.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.648896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.648991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.649085 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.649085 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.649270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] federation.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.649367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.649463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.649569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.649665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.649759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.649862 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.649967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.650063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.650157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.650252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.650348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.650457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.650555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.650651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.650746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.650851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.650946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.651118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.651218 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.651314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.651416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.651512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.651615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.651711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.651813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.651909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.652057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.652162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.652311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.652423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.653065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.653167 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.653271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.653503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.653610 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.653711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.653791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.653864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.653864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.654004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.654078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.654148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.654207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.654331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.654448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.654560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.654666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.654789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.654911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.655033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.655149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.655272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.655387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.655501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.655615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.655730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.655844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.655965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.656081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.656195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.656308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.656422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.656535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.656717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.656838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.656953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.657074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.657189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.657304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.657426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.657543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.657657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.657772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.657887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.658001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.658124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.658239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.658354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.658354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.658576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.658690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.658812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.658929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.password = **** {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.659043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.659156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.659269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.659383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.659504 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.659629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.659745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.659858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.659973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.password = **** {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.660086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.660208 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.661302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.661421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.661520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.661520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.661700 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.661810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.661909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.662985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.663083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.663179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.663274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.663376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.663473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.663579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.663687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.663747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.663920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.663990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.664161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.664277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.664409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.664513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.664674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.664795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.664924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.665040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.665161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.665276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.665390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.665503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.665626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.665741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.665855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.665968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.666082 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.666196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.666318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.666439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.666576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.666694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.666808 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.666921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.667043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.667159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.667274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.667388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.667525 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.667644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.667767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.667882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.667997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.668149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.668268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.668408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.668535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.668669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.668786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.668923 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.669042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.669156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.669282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] policy.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.669399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.669513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.669626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.669740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.669857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.669978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.670094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] receipt.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.670207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.670321 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.670436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.670550 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.670672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.670788 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.670902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] policy.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.671015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.671129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.671293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.671503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.671625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.671739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] resource.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.671852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] receipt.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.671993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.672112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.672235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] resource.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.672351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.672464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.672579 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.673018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.673132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.673258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.673374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.673489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] resource.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.673603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] revoke.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.673717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.673830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.673952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.674067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] resource.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.674181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] role.cache_time = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.674294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.674408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] role.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.674522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.674711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] role.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.674829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.674943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] role.list_limit = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.675056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] revoke.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.675170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.675285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.675407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.675523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.675638 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.675752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] role.cache_time = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.675911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.676039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] role.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.676161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.676277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] role.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.676391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] role.list_limit = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.676512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.676747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.676866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.676995 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.677111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.677226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.677340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.677454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.677576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.677692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.677808 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.677922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.678037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.678174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.678301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.678461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.678618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.678737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.678852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.678966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.679110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.679231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.679366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.679484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.679619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.679738 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.679862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.680089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.680211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.680325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.680445 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.680561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.680708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.680824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.680938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.681051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.681165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.681278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.681400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.681515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.681630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.681744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.681857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.681972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.682094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.682209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.682322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.682436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.682550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.682663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.682785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.682901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] token.cache_time = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.683016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.683129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] token.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.683243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.683364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.683479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.683593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] token.provider = fernet {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.683706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.683821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.683935 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.684057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.684173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.684287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.684401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] token.cache_time = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.684515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.684678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] token.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.684808 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.684924 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.685038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.685153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] token.provider = fernet {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.685266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] trust.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.685417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.685540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.685656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.685770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.685883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.685998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.686112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.686235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.686350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.686464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.686577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.686692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.686805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] trust.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.686928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.687043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.687157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.687272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.687386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.connection = **** {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.687509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.687633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.687749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.687872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.688055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.688199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.688317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.688431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.688545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.688687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.688804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.688918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.689032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.connection = **** {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.689155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.689155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.689383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.689506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.689622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.689736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.689850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.689964 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.690079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.690204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.690322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.690436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.690551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.690665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.690779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.690901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.691017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.691131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.691252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.691368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.692001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.692207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.692386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.692529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=92190) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 29 04:33:00.692662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.692662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.692917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.693162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.693401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.693651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.693896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 29 04:33:00.694072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=92189) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 29 04:33:00.721476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.721704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.721931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.722925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.723167 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.723398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.730371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.731776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.760653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.760883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.761752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.761991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.814216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.815074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.864948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.865178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.866262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.866502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.932295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.932522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.932679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.932767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.932918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.933004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.933148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.933230 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.933372 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.933465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.933595 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.933814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.957894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.958054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.958164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.958294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.958380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.958526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.958609 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.958734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.958817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.958944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.959034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.959155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.959237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.959439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.970411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.970665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.983710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.983947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.984046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.984176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.984259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.984402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.984483 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.984650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.984734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.984875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.984957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.985089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.985169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=92189) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:00.985305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=92190) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 29 04:33:01.002186 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.002399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 29 04:33:01.002594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.002739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.002835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 29 04:33:01.002945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 29 04:33:01.003123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.003203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.003323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 29 04:33:01.003465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.003590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.003724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.003848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.004529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.004742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.005029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.014270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.014554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.014825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.015064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.015147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.015340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.015610 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.015892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.022574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.023365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.027264 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.027546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.027830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.027999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.028114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.028274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.028398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.028551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.028686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.028851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.028943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.029129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.029209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.029384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.029463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.029648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.029758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.029918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.029998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.030183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.030453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.030714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.053463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.054036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.054953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.055237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.055515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.055794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.059842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.060126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.060357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.060439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.060656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.060940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.067727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.067968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.068211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.068490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.072173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.072462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.072570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.072785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.072879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.073071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.073154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.073348 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.073430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.073634 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.073728 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.073918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.073999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.074194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.074274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.074468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.090579 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.090858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.091212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.091489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.094819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.095391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.098021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.098320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.098567 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.098856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.102677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.103181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.105879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.106369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.109064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.109368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.109533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.109659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.109832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.109957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.110117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.110237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.110400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.110534 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.110684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.110971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.127061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.127349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.127619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.127885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.127981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.128204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.128467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.128765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.134977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.135648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.136528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.136829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.137210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.137484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.139648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.139929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.140224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.140307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.140510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.140622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.140824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.140908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.141098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.141381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.152770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.153053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.153237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.153334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.153515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.153597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.153797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.153913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.154080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.154177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.154357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.154452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.154628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.154908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.169896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.170342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.173153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.173551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.176723 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.177282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.179236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.179692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.180826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.181116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.181298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.181403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.181590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.181869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.187891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.188254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.189514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.189871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.192777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.192955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.193069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.193243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.193340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.193523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.193618 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.193794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.201627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.201754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.201915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.202039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.202215 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.202328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.209727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=92189) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.209900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=92190) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 29 04:33:01.209997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.210176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.210285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.210463 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.210571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.210744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.210835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.211015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.211105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.211293 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.211375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.211560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.211643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.211829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.211976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.212197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.212282 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.212482 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.212564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.212789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.212885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.213051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.213132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.213320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.213401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=92189) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.213578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=92190) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 29 04:33:01.246996 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x563d117e19c0 pid: 92189 (default app) Jul 29 04:33:01.247149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x563d117e19c0 pid: 92190 (default app) Jul 29 04:33:01.250356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52782b0-6337-458c-89c4-0e9da610240d None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:01.250653 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52782b0-6337-458c-89c4-0e9da610240d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:01.250935 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52782b0-6337-458c-89c4-0e9da610240d None None] PATH_INFO: `/v3` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:01.255481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1/1] 10.222.0.40 () {50 vars in 791 bytes} [Fri Jul 29 04:33:01 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:09.005118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7285c969-30ab-4068-a263-53adf21ea97c None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:09.005659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7285c969-30ab-4068-a263-53adf21ea97c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:09.005659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7285c969-30ab-4068-a263-53adf21ea97c None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:09.010177 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1/2] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:09 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:33:09.015595 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:09.016051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:09.016146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:09.088594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: INFO dbcounter [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] Registered counter for database keystone Jul 29 04:33:09.110091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG oslo_db.sqlalchemy.engines [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=92189) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 29 04:33:09.117292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writer thread running {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 29 04:33:09.294489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG passlib.handlers.bcrypt [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=92189) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 29 04:33:09.294840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG passlib.handlers.bcrypt [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=92189) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 29 04:33:09.350223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:09.351841 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:09.475718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 2/3] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:09 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 461 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 29 04:33:09.624014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7285c969-30ab-4068-a263-53adf21ea97c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:09.694412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: INFO dbcounter [None req-7285c969-30ab-4068-a263-53adf21ea97c None None] Registered counter for database keystone Jul 29 04:33:09.713509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG oslo_db.sqlalchemy.engines [None req-7285c969-30ab-4068-a263-53adf21ea97c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=92190) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 29 04:33:09.719581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writer thread running {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 29 04:33:09.840405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7285c969-30ab-4068-a263-53adf21ea97c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:09.844705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09f9f8e-bfdf-48d5-a50e-fc6fa22af1a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:09.845339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09f9f8e-bfdf-48d5-a50e-fc6fa22af1a3 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:09.845621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09f9f8e-bfdf-48d5-a50e-fc6fa22af1a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:09.845899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09f9f8e-bfdf-48d5-a50e-fc6fa22af1a3 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:09.854412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09f9f8e-bfdf-48d5-a50e-fc6fa22af1a3 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:09.906299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WARNING keystone.common.rbac_enforcer.enforcer [None req-c09f9f8e-bfdf-48d5-a50e-fc6fa22af1a3 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 29 04:33:09.906586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09f9f8e-bfdf-48d5-a50e-fc6fa22af1a3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:09.917271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 2/4] 10.222.0.40 () {58 vars in 1324 bytes} [Fri Jul 29 04:33:09 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:33:09.923626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:09.938969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fe1f07-7a1c-44ab-a7a7-7eb0310a277d None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:09.943628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a9598d-1d91-416b-aefc-1bd3bab2b7d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:09.944769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a9598d-1d91-416b-aefc-1bd3bab2b7d1 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:09.945063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a9598d-1d91-416b-aefc-1bd3bab2b7d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:09.945346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a9598d-1d91-416b-aefc-1bd3bab2b7d1 admin admin] PATH_INFO: `/v3/projects/25a26196f3ac436c876fd26eb5482e4a` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:09.946853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a9598d-1d91-416b-aefc-1bd3bab2b7d1 admin admin] RBAC: Authorizing `identity:get_project(project_id=25a26196f3ac436c876fd26eb5482e4a)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:09.997490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: WARNING keystone.common.rbac_enforcer.enforcer [None req-52a9598d-1d91-416b-aefc-1bd3bab2b7d1 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 29 04:33:09.997843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a9598d-1d91-416b-aefc-1bd3bab2b7d1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:10.002508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 3/5] 10.222.0.40 () {56 vars in 1207 bytes} [Fri Jul 29 04:33:09 2022] GET /identity/v3/projects/25a26196f3ac436c876fd26eb5482e4a => generated 344 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:11.984647 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62e94c8-9932-415e-93e8-ad3153c91b41 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:11.985149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62e94c8-9932-415e-93e8-ad3153c91b41 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:11.985236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62e94c8-9932-415e-93e8-ad3153c91b41 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:11.990121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 3/6] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:11 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:11.995462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d344782a-8ad1-4ef5-adbc-4be61a3b6990 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:11.995879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d344782a-8ad1-4ef5-adbc-4be61a3b6990 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:11.996028 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d344782a-8ad1-4ef5-adbc-4be61a3b6990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:12.015804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-d344782a-8ad1-4ef5-adbc-4be61a3b6990 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:12.017353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d344782a-8ad1-4ef5-adbc-4be61a3b6990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:12.025195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 4/7] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:11 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Jul 29 04:33:12.046822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62e94c8-9932-415e-93e8-ad3153c91b41 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:12.059606 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62e94c8-9932-415e-93e8-ad3153c91b41 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:12.063840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfef2470-0466-4911-bac1-f25c1c5d4034 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:12.064454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfef2470-0466-4911-bac1-f25c1c5d4034 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:12.064780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfef2470-0466-4911-bac1-f25c1c5d4034 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:12.065066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfef2470-0466-4911-bac1-f25c1c5d4034 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:12.072797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfef2470-0466-4911-bac1-f25c1c5d4034 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:12.074581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfef2470-0466-4911-bac1-f25c1c5d4034 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:12.083660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 4/8] 10.222.0.40 () {58 vars in 1324 bytes} [Fri Jul 29 04:33:12 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:33:12.088986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d344782a-8ad1-4ef5-adbc-4be61a3b6990 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:12.098373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d344782a-8ad1-4ef5-adbc-4be61a3b6990 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:12.102706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb4f930-0f2d-42d0-b87d-6640e7214311 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:12.103340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4f930-0f2d-42d0-b87d-6640e7214311 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:12.103625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4f930-0f2d-42d0-b87d-6640e7214311 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:12.103903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4f930-0f2d-42d0-b87d-6640e7214311 admin admin] PATH_INFO: `/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:12.105283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb4f930-0f2d-42d0-b87d-6640e7214311 admin admin] RBAC: Authorizing `identity:get_user(user_id=cfc9fe47ed4740a0aaddec545dbaca1a)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:12.107091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb4f930-0f2d-42d0-b87d-6640e7214311 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:12.108630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 5/9] 10.222.0.40 () {56 vars in 1198 bytes} [Fri Jul 29 04:33:12 2022] GET /identity/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:14.138367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ba24fa-23f3-4d51-ab89-cc73cc168321 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:14.138900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ba24fa-23f3-4d51-ab89-cc73cc168321 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:14.138900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ba24fa-23f3-4d51-ab89-cc73cc168321 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:14.140241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 5/10] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:33:14.145959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c544b5fc-2fd5-4a55-aa78-54a0d616a81a None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:14.146387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c544b5fc-2fd5-4a55-aa78-54a0d616a81a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:14.146596 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c544b5fc-2fd5-4a55-aa78-54a0d616a81a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:14.167003 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c544b5fc-2fd5-4a55-aa78-54a0d616a81a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:14.168222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c544b5fc-2fd5-4a55-aa78-54a0d616a81a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:14.188283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300a081f-e56a-430d-b47f-08474bc22613 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:14.188582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300a081f-e56a-430d-b47f-08474bc22613 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:14.188896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300a081f-e56a-430d-b47f-08474bc22613 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:14.190222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 6/11] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:33:14.195871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782c46aa-561d-4653-b64c-b673e1db12e1 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:14.196194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782c46aa-561d-4653-b64c-b673e1db12e1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:14.196478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782c46aa-561d-4653-b64c-b673e1db12e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:14.238870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG passlib.handlers.bcrypt [None req-782c46aa-561d-4653-b64c-b673e1db12e1 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=92190) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 29 04:33:14.239308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG passlib.handlers.bcrypt [None req-782c46aa-561d-4653-b64c-b673e1db12e1 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=92190) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 29 04:33:14.251145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 6/12] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:14.274720 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c544b5fc-2fd5-4a55-aa78-54a0d616a81a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:14.285645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-782c46aa-561d-4653-b64c-b673e1db12e1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:14.287294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-782c46aa-561d-4653-b64c-b673e1db12e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:14.297738 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:14.377272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c544b5fc-2fd5-4a55-aa78-54a0d616a81a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:14.387264 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 7/13] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:14.409345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782c46aa-561d-4653-b64c-b673e1db12e1 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:14.411729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a0e44f-9f9c-42ad-b485-b2e02749a7ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:14.412391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a0e44f-9f9c-42ad-b485-b2e02749a7ab None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:14.412776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a0e44f-9f9c-42ad-b485-b2e02749a7ab None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:14.413067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a0e44f-9f9c-42ad-b485-b2e02749a7ab None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:14.424778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a0e44f-9f9c-42ad-b485-b2e02749a7ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-36a0e44f-9f9c-42ad-b485-b2e02749a7ab None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:14.434013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:14.434941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:14.435766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 7/14] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:33:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:14.441047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a0e44f-9f9c-42ad-b485-b2e02749a7ab None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:14.514271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782c46aa-561d-4653-b64c-b673e1db12e1 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:14.516545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:14.525971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:14.544491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a0e44f-9f9c-42ad-b485-b2e02749a7ab None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:14.554365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b341cd0c-9b5e-4eae-b107-171503809ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:14.555234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b341cd0c-9b5e-4eae-b107-171503809ea1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:14.555528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b341cd0c-9b5e-4eae-b107-171503809ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:14.555810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b341cd0c-9b5e-4eae-b107-171503809ea1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:14.578795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b31972-ada5-41a4-8059-67f7d5ef6857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:14.579421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b31972-ada5-41a4-8059-67f7d5ef6857 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:14.579777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b31972-ada5-41a4-8059-67f7d5ef6857 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:14.580057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b31972-ada5-41a4-8059-67f7d5ef6857 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:14.580814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b31972-ada5-41a4-8059-67f7d5ef6857 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:14.581123 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b31972-ada5-41a4-8059-67f7d5ef6857 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:14.583124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b31972-ada5-41a4-8059-67f7d5ef6857 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:14.594703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 8/15] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:14.600463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b31972-ada5-41a4-8059-67f7d5ef6857 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:14.625508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b341cd0c-9b5e-4eae-b107-171503809ea1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:14.627051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b341cd0c-9b5e-4eae-b107-171503809ea1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:14.735142 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 8/16] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 29 04:33:14.741680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b341cd0c-9b5e-4eae-b107-171503809ea1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:14.744886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:14.814231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b31972-ada5-41a4-8059-67f7d5ef6857 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:14.843031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b341cd0c-9b5e-4eae-b107-171503809ea1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:14.848015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d546ef6-b554-43c6-8b4b-0c5cda86a2e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:14.848691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d546ef6-b554-43c6-8b4b-0c5cda86a2e9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:14.848980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d546ef6-b554-43c6-8b4b-0c5cda86a2e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:14.849267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d546ef6-b554-43c6-8b4b-0c5cda86a2e9 None admin] PATH_INFO: `/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:14.850461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d546ef6-b554-43c6-8b4b-0c5cda86a2e9 None admin] RBAC: Authorizing `identity:get_user(user_id=cfc9fe47ed4740a0aaddec545dbaca1a)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:14.851921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d546ef6-b554-43c6-8b4b-0c5cda86a2e9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:14.853555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 9/17] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:14 2022] GET /identity/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a => generated 247 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:14.858850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d546ef6-b554-43c6-8b4b-0c5cda86a2e9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:14.874193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:14.879810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da8de5-7f76-41e5-8ca3-202189103c6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:14.880703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da8de5-7f76-41e5-8ca3-202189103c6d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:14.881000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da8de5-7f76-41e5-8ca3-202189103c6d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:14.881286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43da8de5-7f76-41e5-8ca3-202189103c6d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:14.899930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43da8de5-7f76-41e5-8ca3-202189103c6d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:14.901426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43da8de5-7f76-41e5-8ca3-202189103c6d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-43da8de5-7f76-41e5-8ca3-202189103c6d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:33:14.920258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:14.921226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:14.921993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:33:14.921993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:33:14.921993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:33:14.921993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:14.921993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 9/18] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:33:14.927576 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da8de5-7f76-41e5-8ca3-202189103c6d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:14.961048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d546ef6-b554-43c6-8b4b-0c5cda86a2e9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:14.982353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:14.994971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd52ff16-e1b0-479a-9bec-8dd4b522854d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:14.995552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd52ff16-e1b0-479a-9bec-8dd4b522854d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:14.995836 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd52ff16-e1b0-479a-9bec-8dd4b522854d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:14.996166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd52ff16-e1b0-479a-9bec-8dd4b522854d None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:14.997423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd52ff16-e1b0-479a-9bec-8dd4b522854d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:14.998816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd52ff16-e1b0-479a-9bec-8dd4b522854d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:15.000365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 10/19] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:15.006646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd52ff16-e1b0-479a-9bec-8dd4b522854d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:15.029001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43da8de5-7f76-41e5-8ca3-202189103c6d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:15.062020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf3d24d-1ac9-40a6-8bd7-f3f4fd97f163 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:15.062649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf3d24d-1ac9-40a6-8bd7-f3f4fd97f163 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:15.062941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf3d24d-1ac9-40a6-8bd7-f3f4fd97f163 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:15.063225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf3d24d-1ac9-40a6-8bd7-f3f4fd97f163 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:15.063952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf3d24d-1ac9-40a6-8bd7-f3f4fd97f163 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:15.064272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf3d24d-1ac9-40a6-8bd7-f3f4fd97f163 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:15.065717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf3d24d-1ac9-40a6-8bd7-f3f4fd97f163 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:15.085094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 10/20] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:33:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:15.087988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:15.109162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd52ff16-e1b0-479a-9bec-8dd4b522854d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:15.147274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec761951-428a-421f-8d2a-1c152b229dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:15.147920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec761951-428a-421f-8d2a-1c152b229dc1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:15.148197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec761951-428a-421f-8d2a-1c152b229dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:15.148480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec761951-428a-421f-8d2a-1c152b229dc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:15.149328 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec761951-428a-421f-8d2a-1c152b229dc1 None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.domain.id=default, user.id=cfc9fe47ed4740a0aaddec545dbaca1a) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:15.149649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec761951-428a-421f-8d2a-1c152b229dc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:15.151160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec761951-428a-421f-8d2a-1c152b229dc1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:15.158705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 11/21] 10.222.0.40 () {56 vars in 1322 bytes} [Fri Jul 29 04:33:15 2022] GET /identity/v3/role_assignments?user.id=cfc9fe47ed4740a0aaddec545dbaca1a&scope.domain.id=default&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 233 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:17.446737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ea85f-4e43-45a6-a5c8-cfec1c8670b7 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:17.447751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ea85f-4e43-45a6-a5c8-cfec1c8670b7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:17.448260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ea85f-4e43-45a6-a5c8-cfec1c8670b7 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:17.449849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 11/22] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:17.455159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f968c00-c1db-422a-8557-d52df6232c5e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:17.455971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f968c00-c1db-422a-8557-d52df6232c5e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:17.456449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f968c00-c1db-422a-8557-d52df6232c5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:17.485678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0f968c00-c1db-422a-8557-d52df6232c5e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:17.487340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0f968c00-c1db-422a-8557-d52df6232c5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:17.581584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 12/23] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:17.614411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8ea85f-4e43-45a6-a5c8-cfec1c8670b7 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:17.772836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8ea85f-4e43-45a6-a5c8-cfec1c8670b7 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:17.800179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:17.825502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db60235-47bb-4329-87d4-737589a0cf13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:17.826155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db60235-47bb-4329-87d4-737589a0cf13 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:17.826445 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db60235-47bb-4329-87d4-737589a0cf13 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:17.826727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db60235-47bb-4329-87d4-737589a0cf13 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:17.841435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db60235-47bb-4329-87d4-737589a0cf13 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:17.842940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db60235-47bb-4329-87d4-737589a0cf13 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:17.844076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe209fc-d80b-4e44-b05c-d94f0d6fab97 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:17.844632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe209fc-d80b-4e44-b05c-d94f0d6fab97 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:17.845122 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe209fc-d80b-4e44-b05c-d94f0d6fab97 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:17.848017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 13/24] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:17.852293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3236c-19cb-4498-b1da-4757a06d8a69 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:17.852850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3236c-19cb-4498-b1da-4757a06d8a69 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:17.853352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc3236c-19cb-4498-b1da-4757a06d8a69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-9db60235-47bb-4329-87d4-737589a0cf13 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:33:17.865369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:17.866225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:17.867459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:33:17.867459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:33:17.867459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:33:17.867459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:17.867459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 12/25] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:17.872423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db60235-47bb-4329-87d4-737589a0cf13 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:17.897146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-bdc3236c-19cb-4498-b1da-4757a06d8a69 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:17.899127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-bdc3236c-19cb-4498-b1da-4757a06d8a69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:18.009031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 14/26] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:18.010737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712c3543-f845-4ced-81f2-48cb3f62ff2f None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.011264 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712c3543-f845-4ced-81f2-48cb3f62ff2f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.011788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712c3543-f845-4ced-81f2-48cb3f62ff2f None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.013263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 15/27] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:18.014609 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bb4f7f-8dfc-4a9d-a8ee-169f5c1491cf None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.015156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bb4f7f-8dfc-4a9d-a8ee-169f5c1491cf None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.015631 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bb4f7f-8dfc-4a9d-a8ee-169f5c1491cf None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.017050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 16/28] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:18.026246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f458432f-ae80-4ecc-aa93-5b5e9d69bfb1 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.026870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f458432f-ae80-4ecc-aa93-5b5e9d69bfb1 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.027359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f458432f-ae80-4ecc-aa93-5b5e9d69bfb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.035362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db60235-47bb-4329-87d4-737589a0cf13 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:18.041037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:18.053738 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f458432f-ae80-4ecc-aa93-5b5e9d69bfb1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:18.055565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f458432f-ae80-4ecc-aa93-5b5e9d69bfb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:18.076448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cdd04c-d302-4896-8d44-2286a23cef82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:18.077106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cdd04c-d302-4896-8d44-2286a23cef82 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.077392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cdd04c-d302-4896-8d44-2286a23cef82 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.077670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cdd04c-d302-4896-8d44-2286a23cef82 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.078423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cdd04c-d302-4896-8d44-2286a23cef82 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:18.078730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cdd04c-d302-4896-8d44-2286a23cef82 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:18.080285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cdd04c-d302-4896-8d44-2286a23cef82 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:18.093122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 13/29] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:33:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:18.094477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54968386-ac60-45b1-80d3-86a435f9a281 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.094767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54968386-ac60-45b1-80d3-86a435f9a281 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.095047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54968386-ac60-45b1-80d3-86a435f9a281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.135694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 17/30] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:18.136644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f458432f-ae80-4ecc-aa93-5b5e9d69bfb1 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:18.242399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-54968386-ac60-45b1-80d3-86a435f9a281 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:18.243607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-54968386-ac60-45b1-80d3-86a435f9a281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:18.259197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f458432f-ae80-4ecc-aa93-5b5e9d69bfb1 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:18.268957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:18.296422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4c5902-8d65-408e-9b66-0ada9927221a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:18.297009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4c5902-8d65-408e-9b66-0ada9927221a None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.297299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4c5902-8d65-408e-9b66-0ada9927221a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.297582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4c5902-8d65-408e-9b66-0ada9927221a None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.298585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4c5902-8d65-408e-9b66-0ada9927221a None admin] RBAC: Authorizing `identity:create_role()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:18.300210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4c5902-8d65-408e-9b66-0ada9927221a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:18.304195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:18.320910 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 14/31] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:18.321757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54968386-ac60-45b1-80d3-86a435f9a281 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:18.428879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54968386-ac60-45b1-80d3-86a435f9a281 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:18.464635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 18/32] 10.222.0.40 () {60 vars in 1131 bytes} [Fri Jul 29 04:33:18 2022] POST /identity/v3/roles => generated 219 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:18.465659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4c5902-8d65-408e-9b66-0ada9927221a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:18.492303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951a375d-0a8b-40f5-9570-8bd1483de91d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:18.494183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951a375d-0a8b-40f5-9570-8bd1483de91d None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.497587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951a375d-0a8b-40f5-9570-8bd1483de91d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.497900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951a375d-0a8b-40f5-9570-8bd1483de91d None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.498876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951a375d-0a8b-40f5-9570-8bd1483de91d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:18.500380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951a375d-0a8b-40f5-9570-8bd1483de91d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:18.621371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4c5902-8d65-408e-9b66-0ada9927221a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:18.692694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ac8600-1edc-4c1e-9741-ca67d34845eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:18.693445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ac8600-1edc-4c1e-9741-ca67d34845eb None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.693734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ac8600-1edc-4c1e-9741-ca67d34845eb None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.694013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ac8600-1edc-4c1e-9741-ca67d34845eb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.705046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ac8600-1edc-4c1e-9741-ca67d34845eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-60ac8600-1edc-4c1e-9741-ca67d34845eb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:18.710660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:18.711659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:18.712458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 19/33] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:33:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:18.713115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ac8600-1edc-4c1e-9741-ca67d34845eb None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:18.744130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 15/34] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:33:18 2022] POST /identity/v3/projects => generated 301 bytes in 423 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:18.745392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951a375d-0a8b-40f5-9570-8bd1483de91d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:18.773543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:18.804744 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:18.876222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ac8600-1edc-4c1e-9741-ca67d34845eb None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:18.908730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951a375d-0a8b-40f5-9570-8bd1483de91d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:18.929543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5742e549-7620-4ec9-a924-c18906322d3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:18.930188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5742e549-7620-4ec9-a924-c18906322d3f None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.930481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5742e549-7620-4ec9-a924-c18906322d3f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.930766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5742e549-7620-4ec9-a924-c18906322d3f None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.931982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5742e549-7620-4ec9-a924-c18906322d3f None admin] RBAC: Authorizing `identity:create_role()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:18.933914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5742e549-7620-4ec9-a924-c18906322d3f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:18.954266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837404eb-feb9-40f1-b131-aa63e1d167eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:18.954922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837404eb-feb9-40f1-b131-aa63e1d167eb None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.955061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 20/35] 10.222.0.40 () {60 vars in 1131 bytes} [Fri Jul 29 04:33:18 2022] POST /identity/v3/roles => generated 225 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:18.955718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837404eb-feb9-40f1-b131-aa63e1d167eb None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.956067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837404eb-feb9-40f1-b131-aa63e1d167eb None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.956419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47760db-fdb1-4005-a101-2e07d2a81b2e None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.956842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47760db-fdb1-4005-a101-2e07d2a81b2e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.957526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837404eb-feb9-40f1-b131-aa63e1d167eb None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:18.958216 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47760db-fdb1-4005-a101-2e07d2a81b2e None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.958320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837404eb-feb9-40f1-b131-aa63e1d167eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:18.959112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 21/36] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:18.961267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837404eb-feb9-40f1-b131-aa63e1d167eb None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:18.964546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f49d7-c5d7-4f59-b3e3-526c7956c2a4 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:18.964873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f49d7-c5d7-4f59-b3e3-526c7956c2a4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:18.965152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f49d7-c5d7-4f59-b3e3-526c7956c2a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:18.986138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 16/37] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:18.994424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837404eb-feb9-40f1-b131-aa63e1d167eb None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:18.996037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f18f49d7-c5d7-4f59-b3e3-526c7956c2a4 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:18.998363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f18f49d7-c5d7-4f59-b3e3-526c7956c2a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:19.090494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 22/38] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:19.092243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:19.117878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18f49d7-c5d7-4f59-b3e3-526c7956c2a4 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:19.127375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837404eb-feb9-40f1-b131-aa63e1d167eb None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:19.168855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34af9b63-ed11-47ef-9d14-dd5f2c0b25b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:19.169779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34af9b63-ed11-47ef-9d14-dd5f2c0b25b0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:19.170283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34af9b63-ed11-47ef-9d14-dd5f2c0b25b0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:19.170766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34af9b63-ed11-47ef-9d14-dd5f2c0b25b0 None admin] PATH_INFO: `/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:19.173485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34af9b63-ed11-47ef-9d14-dd5f2c0b25b0 None admin] RBAC: Authorizing `identity:get_user(user_id=cfc9fe47ed4740a0aaddec545dbaca1a)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:19.175197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34af9b63-ed11-47ef-9d14-dd5f2c0b25b0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:19.177522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 17/39] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:18 2022] GET /identity/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a => generated 247 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:19.178785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a883cad4-0b87-4446-a861-d10309671738 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:19.179081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a883cad4-0b87-4446-a861-d10309671738 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:19.179367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a883cad4-0b87-4446-a861-d10309671738 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:19.180582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 18/40] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:19.183433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a883cad4-0b87-4446-a861-d10309671738 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:19.187014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:19.249687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18f49d7-c5d7-4f59-b3e3-526c7956c2a4 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:19.284964 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4535a286-70a7-4b53-8983-9c8eafe94851 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:19.285681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4535a286-70a7-4b53-8983-9c8eafe94851 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:19.285976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4535a286-70a7-4b53-8983-9c8eafe94851 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:19.286269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4535a286-70a7-4b53-8983-9c8eafe94851 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:19.286993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4535a286-70a7-4b53-8983-9c8eafe94851 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:19.288399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4535a286-70a7-4b53-8983-9c8eafe94851 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:19.292957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a883cad4-0b87-4446-a861-d10309671738 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:19.303268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 23/41] 10.222.0.40 () {60 vars in 1131 bytes} [Fri Jul 29 04:33:19 2022] POST /identity/v3/roles => generated 223 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:33:19.305978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577999dd-146f-48c6-a060-f18ac9ac038c None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:19.306278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577999dd-146f-48c6-a060-f18ac9ac038c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:19.306569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577999dd-146f-48c6-a060-f18ac9ac038c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:19.326791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-577999dd-146f-48c6-a060-f18ac9ac038c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:19.328055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-577999dd-146f-48c6-a060-f18ac9ac038c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:19.331888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3a5deb-3d8e-493a-9cd5-f9a90191f0a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:19.332780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3a5deb-3d8e-493a-9cd5-f9a90191f0a4 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:19.332910 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3a5deb-3d8e-493a-9cd5-f9a90191f0a4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:19.333185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3a5deb-3d8e-493a-9cd5-f9a90191f0a4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:19.334529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3a5deb-3d8e-493a-9cd5-f9a90191f0a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:19.336107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3a5deb-3d8e-493a-9cd5-f9a90191f0a4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:19.338140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 19/42] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:19.345848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3a5deb-3d8e-493a-9cd5-f9a90191f0a4 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:19.359800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:19.420716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 24/43] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:19.434607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577999dd-146f-48c6-a060-f18ac9ac038c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:19.462670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3a5deb-3d8e-493a-9cd5-f9a90191f0a4 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:19.474493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:19.503867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415f59a6-ab16-4b85-a9a0-57e02c44f939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:19.504820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415f59a6-ab16-4b85-a9a0-57e02c44f939 None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:19.505120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415f59a6-ab16-4b85-a9a0-57e02c44f939 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:19.505412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415f59a6-ab16-4b85-a9a0-57e02c44f939 None admin] PATH_INFO: `/v3/domains/default/users/cfc9fe47ed4740a0aaddec545dbaca1a/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:19.507456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415f59a6-ab16-4b85-a9a0-57e02c44f939 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=cfc9fe47ed4740a0aaddec545dbaca1a, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:19.508967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415f59a6-ab16-4b85-a9a0-57e02c44f939 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:19.525353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 20/44] 10.222.0.40 () {58 vars in 1361 bytes} [Fri Jul 29 04:33:19 2022] PUT /identity/v3/domains/default/users/cfc9fe47ed4740a0aaddec545dbaca1a/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:33:19.565961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577999dd-146f-48c6-a060-f18ac9ac038c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:19.603566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd45cbfc-54de-45c3-a759-58bff4f43933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:19.604240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45cbfc-54de-45c3-a759-58bff4f43933 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:19.604544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45cbfc-54de-45c3-a759-58bff4f43933 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:19.604855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd45cbfc-54de-45c3-a759-58bff4f43933 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:19.606098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd45cbfc-54de-45c3-a759-58bff4f43933 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:19.607606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd45cbfc-54de-45c3-a759-58bff4f43933 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:19.609558 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 25/45] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:19.615754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415f59a6-ab16-4b85-a9a0-57e02c44f939 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:19.689007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:19.724059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415f59a6-ab16-4b85-a9a0-57e02c44f939 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:19.757114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bf3e79-1283-4cda-a714-de2675a62d36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:19.757741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf3e79-1283-4cda-a714-de2675a62d36 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:19.758032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf3e79-1283-4cda-a714-de2675a62d36 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:19.758311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf3e79-1283-4cda-a714-de2675a62d36 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:19.759010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bf3e79-1283-4cda-a714-de2675a62d36 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:19.760461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bf3e79-1283-4cda-a714-de2675a62d36 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:19.787279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 21/46] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:33:19 2022] POST /identity/v3/projects => generated 312 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:21.511084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f649d2-b0af-4949-b490-eeec364c1d34 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:21.511603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f649d2-b0af-4949-b490-eeec364c1d34 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:21.511685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f649d2-b0af-4949-b490-eeec364c1d34 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:21.513065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 26/47] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:21.518343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42133c04-33a6-4a0c-b8d6-ea4109127e5c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:21.518737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42133c04-33a6-4a0c-b8d6-ea4109127e5c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:21.518916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42133c04-33a6-4a0c-b8d6-ea4109127e5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:21.537016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-42133c04-33a6-4a0c-b8d6-ea4109127e5c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:21.538556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-42133c04-33a6-4a0c-b8d6-ea4109127e5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:21.605593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 22/48] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:21.629727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f649d2-b0af-4949-b490-eeec364c1d34 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:21.731658 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f649d2-b0af-4949-b490-eeec364c1d34 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:21.733218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:21.766328 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102964bc-d87f-44c9-97ff-54898b404f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:21.766901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102964bc-d87f-44c9-97ff-54898b404f8c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:21.767192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102964bc-d87f-44c9-97ff-54898b404f8c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:21.767479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102964bc-d87f-44c9-97ff-54898b404f8c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:21.777764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102964bc-d87f-44c9-97ff-54898b404f8c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:21.780544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55caed-b429-4193-a6c9-ec7400335b12 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:21.780869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55caed-b429-4193-a6c9-ec7400335b12 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:21.781154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca55caed-b429-4193-a6c9-ec7400335b12 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:21.782533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 23/49] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-102964bc-d87f-44c9-97ff-54898b404f8c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:21.782974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:21.783971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:21.784731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 27/50] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:33:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:21.788129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab74a3-eba2-4a37-bcb1-96ba3b9a8cf5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:21.788424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab74a3-eba2-4a37-bcb1-96ba3b9a8cf5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:21.788743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab74a3-eba2-4a37-bcb1-96ba3b9a8cf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:21.790406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102964bc-d87f-44c9-97ff-54898b404f8c None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:21.807368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-37ab74a3-eba2-4a37-bcb1-96ba3b9a8cf5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:21.808591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-37ab74a3-eba2-4a37-bcb1-96ba3b9a8cf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:21.854371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:21.875844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 24/51] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:21.890646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102964bc-d87f-44c9-97ff-54898b404f8c None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:21.898298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ab74a3-eba2-4a37-bcb1-96ba3b9a8cf5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:21.927181 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8df1ff-bb4e-4c59-bcc9-d1769b7fddbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:21.927798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8df1ff-bb4e-4c59-bcc9-d1769b7fddbd None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:21.928083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8df1ff-bb4e-4c59-bcc9-d1769b7fddbd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:21.928360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8df1ff-bb4e-4c59-bcc9-d1769b7fddbd None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:21.929122 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8df1ff-bb4e-4c59-bcc9-d1769b7fddbd None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:21.929430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8df1ff-bb4e-4c59-bcc9-d1769b7fddbd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:21.930814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8df1ff-bb4e-4c59-bcc9-d1769b7fddbd None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:21.939032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 28/52] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:21.944592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8df1ff-bb4e-4c59-bcc9-d1769b7fddbd None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:21.968756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:21.997702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ab74a3-eba2-4a37-bcb1-96ba3b9a8cf5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:22.031512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdc31f-d717-4d54-9538-404012102d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:22.032089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdc31f-d717-4d54-9538-404012102d39 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:22.032382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdc31f-d717-4d54-9538-404012102d39 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:22.032709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdc31f-d717-4d54-9538-404012102d39 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:22.033887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbdc31f-d717-4d54-9538-404012102d39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:22.036076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbdc31f-d717-4d54-9538-404012102d39 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:22.037866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 25/53] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:22.043177 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdc31f-d717-4d54-9538-404012102d39 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:22.043463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8df1ff-bb4e-4c59-bcc9-d1769b7fddbd None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:22.077175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c44d159-f2d4-457c-909a-b739fd0e6a6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:22.077821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c44d159-f2d4-457c-909a-b739fd0e6a6a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:22.078112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c44d159-f2d4-457c-909a-b739fd0e6a6a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:22.078394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c44d159-f2d4-457c-909a-b739fd0e6a6a None admin] PATH_INFO: `/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:22.079572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c44d159-f2d4-457c-909a-b739fd0e6a6a None admin] RBAC: Authorizing `identity:get_user(user_id=cfc9fe47ed4740a0aaddec545dbaca1a)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:22.081050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c44d159-f2d4-457c-909a-b739fd0e6a6a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:22.083379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 29/54] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:21 2022] GET /identity/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a => generated 247 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:22.088669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c44d159-f2d4-457c-909a-b739fd0e6a6a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:22.094102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:22.145300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdc31f-d717-4d54-9538-404012102d39 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:22.177856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:22.181527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd913d7-a75e-4e1a-8149-fe96926f2f29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:22.182142 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd913d7-a75e-4e1a-8149-fe96926f2f29 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:22.182434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd913d7-a75e-4e1a-8149-fe96926f2f29 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:22.182787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd913d7-a75e-4e1a-8149-fe96926f2f29 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:22.183494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd913d7-a75e-4e1a-8149-fe96926f2f29 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:22.185043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd913d7-a75e-4e1a-8149-fe96926f2f29 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:22.186673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c44d159-f2d4-457c-909a-b739fd0e6a6a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:22.213577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 26/55] 10.222.0.40 () {60 vars in 1140 bytes} [Fri Jul 29 04:33:22 2022] POST /identity/v3/projects => generated 298 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:33:22.224322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d4d56f-182b-4346-a423-039580240f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:22.225333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d4d56f-182b-4346-a423-039580240f2a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:22.225755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d4d56f-182b-4346-a423-039580240f2a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:22.226094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d4d56f-182b-4346-a423-039580240f2a None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:22.227885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d4d56f-182b-4346-a423-039580240f2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:22.229857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d4d56f-182b-4346-a423-039580240f2a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:22.232132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 30/56] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:22.239060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd913d7-a75e-4e1a-8149-fe96926f2f29 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:22.342809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:22.346186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd913d7-a75e-4e1a-8149-fe96926f2f29 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:22.383434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a339428-4ac6-42cd-8745-6caefc55915d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:22.384064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a339428-4ac6-42cd-8745-6caefc55915d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:22.384359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a339428-4ac6-42cd-8745-6caefc55915d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:22.384675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a339428-4ac6-42cd-8745-6caefc55915d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:22.385485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a339428-4ac6-42cd-8745-6caefc55915d None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.domain.id=default, user.id=cfc9fe47ed4740a0aaddec545dbaca1a) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:22.385811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a339428-4ac6-42cd-8745-6caefc55915d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:22.387272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a339428-4ac6-42cd-8745-6caefc55915d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:22.395448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 27/57] 10.222.0.40 () {56 vars in 1322 bytes} [Fri Jul 29 04:33:22 2022] GET /identity/v3/role_assignments?user.id=cfc9fe47ed4740a0aaddec545dbaca1a&scope.domain.id=default&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 530 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:24.241997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5681a59-b267-4b80-8e93-84748d376ada None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:24.242515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5681a59-b267-4b80-8e93-84748d376ada None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:24.242622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5681a59-b267-4b80-8e93-84748d376ada None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:24.243923 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 31/58] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:24.249057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf6e5f8-1eed-41d7-ab61-3f893acd24ca None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:24.249417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf6e5f8-1eed-41d7-ab61-3f893acd24ca None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:24.249616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf6e5f8-1eed-41d7-ab61-3f893acd24ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:24.268102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7cf6e5f8-1eed-41d7-ab61-3f893acd24ca None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:24.270012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7cf6e5f8-1eed-41d7-ab61-3f893acd24ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:24.335156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 28/59] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:24.358323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5681a59-b267-4b80-8e93-84748d376ada None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:24.459608 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5681a59-b267-4b80-8e93-84748d376ada None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:24.493864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa01bc56-db7b-4447-afd2-d75f5930138c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:24.494444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa01bc56-db7b-4447-afd2-d75f5930138c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:24.494738 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa01bc56-db7b-4447-afd2-d75f5930138c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:24.495022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa01bc56-db7b-4447-afd2-d75f5930138c None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:24.496187 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa01bc56-db7b-4447-afd2-d75f5930138c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:24.497680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa01bc56-db7b-4447-afd2-d75f5930138c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:24.499250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 32/60] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:24.504682 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf6e5f8-1eed-41d7-ab61-3f893acd24ca None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:24.595875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:24.603289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf6e5f8-1eed-41d7-ab61-3f893acd24ca None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:24.636017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1523b6e1-732e-4228-aca6-57a60ca66d6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:24.636652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1523b6e1-732e-4228-aca6-57a60ca66d6e None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:24.636952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1523b6e1-732e-4228-aca6-57a60ca66d6e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:24.637235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1523b6e1-732e-4228-aca6-57a60ca66d6e None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:24.637917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1523b6e1-732e-4228-aca6-57a60ca66d6e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:24.639346 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1523b6e1-732e-4228-aca6-57a60ca66d6e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:24.666332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 29/61] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:33:24 2022] POST /identity/v3/users => generated 275 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:27.787286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03366266-f961-49f0-9382-a2b5e53acd08 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:27.787813 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03366266-f961-49f0-9382-a2b5e53acd08 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:27.787888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03366266-f961-49f0-9382-a2b5e53acd08 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:27.789234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 33/62] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:27.794309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2dc392-b470-4486-af1a-f174a1ef1e0c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:27.795121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2dc392-b470-4486-af1a-f174a1ef1e0c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:27.795629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2dc392-b470-4486-af1a-f174a1ef1e0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:27.825639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7f2dc392-b470-4486-af1a-f174a1ef1e0c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:27.827225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7f2dc392-b470-4486-af1a-f174a1ef1e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:27.922108 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7ab6a-e6f8-48be-a6af-d7b71f24aaa8 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:27.922714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7ab6a-e6f8-48be-a6af-d7b71f24aaa8 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:27.923502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7ab6a-e6f8-48be-a6af-d7b71f24aaa8 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:27.925093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 34/63] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:27.926689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 30/64] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:27.940497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7028323-600d-4a59-8307-e4f022bdeed1 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:27.941652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7028323-600d-4a59-8307-e4f022bdeed1 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:27.942160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7028323-600d-4a59-8307-e4f022bdeed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:27.952039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2dc392-b470-4486-af1a-f174a1ef1e0c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:27.967264 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c7028323-600d-4a59-8307-e4f022bdeed1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:27.968471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c7028323-600d-4a59-8307-e4f022bdeed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:28.016975 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:28.070905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 35/65] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:28.072649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cde14b-14cd-4a2e-9497-b53c7cdc6fd4 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.081036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cde14b-14cd-4a2e-9497-b53c7cdc6fd4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.081597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cde14b-14cd-4a2e-9497-b53c7cdc6fd4 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.083405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 36/66] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:28 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:28.089357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0393d1-47f0-4d27-b712-e2af2c9d88e8 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.089883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0393d1-47f0-4d27-b712-e2af2c9d88e8 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.090337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0393d1-47f0-4d27-b712-e2af2c9d88e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.100624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:28.118974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-7f0393d1-47f0-4d27-b712-e2af2c9d88e8 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:28.120316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2dc392-b470-4486-af1a-f174a1ef1e0c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:28.120689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7f0393d1-47f0-4d27-b712-e2af2c9d88e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:28.159717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b30f9c5-5fe4-47b4-90bd-b6074f180454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:28.160535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b30f9c5-5fe4-47b4-90bd-b6074f180454 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.160783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b30f9c5-5fe4-47b4-90bd-b6074f180454 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.161075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b30f9c5-5fe4-47b4-90bd-b6074f180454 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.174185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b30f9c5-5fe4-47b4-90bd-b6074f180454 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-4b30f9c5-5fe4-47b4-90bd-b6074f180454 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:28.181776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:28.182712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:28.183458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 31/67] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:33:28.184083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b30f9c5-5fe4-47b4-90bd-b6074f180454 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:28.199481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 37/68] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:28.200645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2962ba-03c9-440b-b9ef-f46744d517c9 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.201124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2962ba-03c9-440b-b9ef-f46744d517c9 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.201552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2962ba-03c9-440b-b9ef-f46744d517c9 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.203061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 38/69] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:28.207981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2962ba-03c9-440b-b9ef-f46744d517c9 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:28.295050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b30f9c5-5fe4-47b4-90bd-b6074f180454 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:28.308232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:28.323910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2962ba-03c9-440b-b9ef-f46744d517c9 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:28.330362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7078e1-956b-4df9-91a2-90a3422c3108 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:28.331018 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7078e1-956b-4df9-91a2-90a3422c3108 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.331312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7078e1-956b-4df9-91a2-90a3422c3108 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.331641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7078e1-956b-4df9-91a2-90a3422c3108 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.339452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:28.343588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7078e1-956b-4df9-91a2-90a3422c3108 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-9e7078e1-956b-4df9-91a2-90a3422c3108 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:28.349041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:28.349879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:28.350616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 32/70] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:33:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:28.351712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2678220-0007-4cd5-9f35-87e006efb3d3 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.352186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2678220-0007-4cd5-9f35-87e006efb3d3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.352282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2678220-0007-4cd5-9f35-87e006efb3d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.366705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413102ad-95ac-4b3c-868a-cb16b3b8f0a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:28.367336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413102ad-95ac-4b3c-868a-cb16b3b8f0a9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.367625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413102ad-95ac-4b3c-868a-cb16b3b8f0a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.367904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413102ad-95ac-4b3c-868a-cb16b3b8f0a9 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.368643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413102ad-95ac-4b3c-868a-cb16b3b8f0a9 None admin] RBAC: Adding query filter params (name=member) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:28.368957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413102ad-95ac-4b3c-868a-cb16b3b8f0a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:28.370347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413102ad-95ac-4b3c-868a-cb16b3b8f0a9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:28.370921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d2678220-0007-4cd5-9f35-87e006efb3d3 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:28.372129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d2678220-0007-4cd5-9f35-87e006efb3d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:28.379277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 39/71] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:28.381483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413102ad-95ac-4b3c-868a-cb16b3b8f0a9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:28.448687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 33/72] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:28.453598 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2678220-0007-4cd5-9f35-87e006efb3d3 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:28.486038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413102ad-95ac-4b3c-868a-cb16b3b8f0a9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:28.519220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2fa0e1-722b-4f54-aca2-07c3ed89088a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:28.519874 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2fa0e1-722b-4f54-aca2-07c3ed89088a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.520162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2fa0e1-722b-4f54-aca2-07c3ed89088a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.520447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2fa0e1-722b-4f54-aca2-07c3ed89088a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.530903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2fa0e1-722b-4f54-aca2-07c3ed89088a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-cd2fa0e1-722b-4f54-aca2-07c3ed89088a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:28.537456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:28.538440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:28.539185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 40/73] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:33:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:28.541389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2fa0e1-722b-4f54-aca2-07c3ed89088a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:28.554411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:28.556935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2678220-0007-4cd5-9f35-87e006efb3d3 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:28.585569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:28.591777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84add233-8d75-46a8-bf2f-e2ca8ec9fa8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:28.592403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84add233-8d75-46a8-bf2f-e2ca8ec9fa8b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.592716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84add233-8d75-46a8-bf2f-e2ca8ec9fa8b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.593000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84add233-8d75-46a8-bf2f-e2ca8ec9fa8b None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.593714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84add233-8d75-46a8-bf2f-e2ca8ec9fa8b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:28.594020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84add233-8d75-46a8-bf2f-e2ca8ec9fa8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:28.595404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84add233-8d75-46a8-bf2f-e2ca8ec9fa8b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:28.603870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 34/74] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:33:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:28.603870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbfadcb-c162-4793-893a-752aba63e6d7 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.603870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbfadcb-c162-4793-893a-752aba63e6d7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.603870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcbfadcb-c162-4793-893a-752aba63e6d7 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.603870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 35/75] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:28.609371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbfadcb-c162-4793-893a-752aba63e6d7 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:28.643512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2fa0e1-722b-4f54-aca2-07c3ed89088a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:28.678949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dd200e-49e1-49b6-b892-0a28df418177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:28.679606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dd200e-49e1-49b6-b892-0a28df418177 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.679917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dd200e-49e1-49b6-b892-0a28df418177 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.680189 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dd200e-49e1-49b6-b892-0a28df418177 None admin] PATH_INFO: `/v3/users/0c530b748ab741c89390231f1b4f34c9` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.694136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dd200e-49e1-49b6-b892-0a28df418177 None admin] RBAC: Authorizing `identity:get_user(user_id=0c530b748ab741c89390231f1b4f34c9)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:28.695627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dd200e-49e1-49b6-b892-0a28df418177 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:28.697304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 41/76] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:28 2022] GET /identity/v3/users/0c530b748ab741c89390231f1b4f34c9 => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:28.702624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dd200e-49e1-49b6-b892-0a28df418177 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:28.712857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcbfadcb-c162-4793-893a-752aba63e6d7 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:28.746346 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c89d98-d3b3-46da-a87f-fa8f94a9c714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:28.747003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c89d98-d3b3-46da-a87f-fa8f94a9c714 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.747295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c89d98-d3b3-46da-a87f-fa8f94a9c714 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.747581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c89d98-d3b3-46da-a87f-fa8f94a9c714 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.758339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27c89d98-d3b3-46da-a87f-fa8f94a9c714 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-27c89d98-d3b3-46da-a87f-fa8f94a9c714 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:28.763663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:28.764727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:28.765511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 36/77] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:28.766162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c89d98-d3b3-46da-a87f-fa8f94a9c714 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:28.770790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:28.806315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dd200e-49e1-49b6-b892-0a28df418177 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:28.819981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:28.842036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d03d698-43c6-4eba-8b5d-35b871aa7b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:28.842672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d03d698-43c6-4eba-8b5d-35b871aa7b8f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.842965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d03d698-43c6-4eba-8b5d-35b871aa7b8f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.843252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d03d698-43c6-4eba-8b5d-35b871aa7b8f None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.843978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d03d698-43c6-4eba-8b5d-35b871aa7b8f None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:28.844297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d03d698-43c6-4eba-8b5d-35b871aa7b8f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:28.845788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d03d698-43c6-4eba-8b5d-35b871aa7b8f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:28.853764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 42/78] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:28.855075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a0437d-a8c4-4c23-87aa-613edef05e78 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.855375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a0437d-a8c4-4c23-87aa-613edef05e78 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.855660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a0437d-a8c4-4c23-87aa-613edef05e78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.869337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c89d98-d3b3-46da-a87f-fa8f94a9c714 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:28.873901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-52a0437d-a8c4-4c23-87aa-613edef05e78 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:28.875150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-52a0437d-a8c4-4c23-87aa-613edef05e78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:28.903624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6b1e55-5ab5-4750-9c62-640ce121f3d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:28.904275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6b1e55-5ab5-4750-9c62-640ce121f3d9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:28.904567 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6b1e55-5ab5-4750-9c62-640ce121f3d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:28.904883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6b1e55-5ab5-4750-9c62-640ce121f3d9 None admin] PATH_INFO: `/v3/users/0c530b748ab741c89390231f1b4f34c9` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:28.906079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6b1e55-5ab5-4750-9c62-640ce121f3d9 None admin] RBAC: Authorizing `identity:get_user(user_id=0c530b748ab741c89390231f1b4f34c9)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:28.907548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6b1e55-5ab5-4750-9c62-640ce121f3d9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:28.909222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 37/79] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:28 2022] GET /identity/v3/users/0c530b748ab741c89390231f1b4f34c9 => generated 275 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:28.910055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6b1e55-5ab5-4750-9c62-640ce121f3d9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:28.943440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 43/80] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:28.945539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a0437d-a8c4-4c23-87aa-613edef05e78 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:29.003119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:29.014797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6b1e55-5ab5-4750-9c62-640ce121f3d9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:29.034628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:29.043417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a0437d-a8c4-4c23-87aa-613edef05e78 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:29.049807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd3b23a-2746-457a-8020-0ad9f2b8fa54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:29.050973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd3b23a-2746-457a-8020-0ad9f2b8fa54 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:29.051262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd3b23a-2746-457a-8020-0ad9f2b8fa54 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:29.051541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd3b23a-2746-457a-8020-0ad9f2b8fa54 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:29.077963 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b5ab65-7579-4abe-878e-09a0830e6859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:29.078601 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b5ab65-7579-4abe-878e-09a0830e6859 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:29.078878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b5ab65-7579-4abe-878e-09a0830e6859 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:29.079161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b5ab65-7579-4abe-878e-09a0830e6859 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:29.079881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b5ab65-7579-4abe-878e-09a0830e6859 None admin] RBAC: Adding query filter params (name=member) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:29.080192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b5ab65-7579-4abe-878e-09a0830e6859 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:29.081620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b5ab65-7579-4abe-878e-09a0830e6859 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:29.089819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 44/81] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:29.090624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b5ab65-7579-4abe-878e-09a0830e6859 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:29.118516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd3b23a-2746-457a-8020-0ad9f2b8fa54 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:29.120011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd3b23a-2746-457a-8020-0ad9f2b8fa54 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:29.191506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b5ab65-7579-4abe-878e-09a0830e6859 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:29.213773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:29.217801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 38/82] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:29.217801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd3b23a-2746-457a-8020-0ad9f2b8fa54 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:29.226432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555e3eb1-6306-48d2-8fa4-e75ea197b339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:29.227085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555e3eb1-6306-48d2-8fa4-e75ea197b339 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:29.227375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555e3eb1-6306-48d2-8fa4-e75ea197b339 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:29.227664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555e3eb1-6306-48d2-8fa4-e75ea197b339 None admin] PATH_INFO: `/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:29.228910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555e3eb1-6306-48d2-8fa4-e75ea197b339 None admin] RBAC: Authorizing `identity:get_user(user_id=cfc9fe47ed4740a0aaddec545dbaca1a)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:29.230395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555e3eb1-6306-48d2-8fa4-e75ea197b339 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:29.232013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 45/83] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:29 2022] GET /identity/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:29.232795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555e3eb1-6306-48d2-8fa4-e75ea197b339 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:29.243232 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:29.325603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd3b23a-2746-457a-8020-0ad9f2b8fa54 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:29.330229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555e3eb1-6306-48d2-8fa4-e75ea197b339 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:29.359157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a494b0-fa5f-42c9-964d-42106bc36bdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:29.359778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a494b0-fa5f-42c9-964d-42106bc36bdf None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:29.360074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a494b0-fa5f-42c9-964d-42106bc36bdf None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:29.360358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a494b0-fa5f-42c9-964d-42106bc36bdf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:29.362382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb141b2-23ca-4411-b385-af1660a5e30e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:29.363627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb141b2-23ca-4411-b385-af1660a5e30e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:29.363912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb141b2-23ca-4411-b385-af1660a5e30e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:29.364191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb141b2-23ca-4411-b385-af1660a5e30e None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:29.365375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb141b2-23ca-4411-b385-af1660a5e30e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:29.366814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb141b2-23ca-4411-b385-af1660a5e30e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:29.368383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 46/84] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:29.370017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb141b2-23ca-4411-b385-af1660a5e30e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:29.427025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:29.427329 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a494b0-fa5f-42c9-964d-42106bc36bdf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:29.428852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a494b0-fa5f-42c9-964d-42106bc36bdf None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:29.449774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:29.468436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb141b2-23ca-4411-b385-af1660a5e30e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:29.503060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e646812-4e51-47d7-8f6a-044efce151da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:29.504631 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e646812-4e51-47d7-8f6a-044efce151da None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:29.504945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e646812-4e51-47d7-8f6a-044efce151da None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:29.505233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e646812-4e51-47d7-8f6a-044efce151da None admin] PATH_INFO: `/v3/users/0c530b748ab741c89390231f1b4f34c9` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:29.506752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e646812-4e51-47d7-8f6a-044efce151da None admin] RBAC: Authorizing `identity:get_user(user_id=0c530b748ab741c89390231f1b4f34c9)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:29.508207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e646812-4e51-47d7-8f6a-044efce151da None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:29.509890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 47/85] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:29 2022] GET /identity/v3/users/0c530b748ab741c89390231f1b4f34c9 => generated 275 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:29.514391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e646812-4e51-47d7-8f6a-044efce151da None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:29.533409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 39/86] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:29.536384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a494b0-fa5f-42c9-964d-42106bc36bdf None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:29.612383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e646812-4e51-47d7-8f6a-044efce151da None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:29.632674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a494b0-fa5f-42c9-964d-42106bc36bdf None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:29.638816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:29.649338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14eb7fc-1d15-4859-a7eb-aa33868010f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:29.649988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14eb7fc-1d15-4859-a7eb-aa33868010f5 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:29.650278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14eb7fc-1d15-4859-a7eb-aa33868010f5 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:29.650559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14eb7fc-1d15-4859-a7eb-aa33868010f5 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:29.651727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14eb7fc-1d15-4859-a7eb-aa33868010f5 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:29.653227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14eb7fc-1d15-4859-a7eb-aa33868010f5 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:29.654335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:29.654891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 48/87] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:29 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:29.656070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14eb7fc-1d15-4859-a7eb-aa33868010f5 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:29.669976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c898e2e-53d5-4462-8b52-ccb686dfe90c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:29.670637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c898e2e-53d5-4462-8b52-ccb686dfe90c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:29.670928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c898e2e-53d5-4462-8b52-ccb686dfe90c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:29.671208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c898e2e-53d5-4462-8b52-ccb686dfe90c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:29.739592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c898e2e-53d5-4462-8b52-ccb686dfe90c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:29.741099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c898e2e-53d5-4462-8b52-ccb686dfe90c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:29.757457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14eb7fc-1d15-4859-a7eb-aa33868010f5 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:29.790064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01c0e99-dab1-410c-b08d-0f66d6da4489 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:29.790683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01c0e99-dab1-410c-b08d-0f66d6da4489 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:29.790966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01c0e99-dab1-410c-b08d-0f66d6da4489 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:29.791242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01c0e99-dab1-410c-b08d-0f66d6da4489 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:29.791924 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01c0e99-dab1-410c-b08d-0f66d6da4489 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:29.793399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01c0e99-dab1-410c-b08d-0f66d6da4489 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:29.821226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 49/88] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:33:29 2022] POST /identity/v3/users => generated 289 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:29.822051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01c0e99-dab1-410c-b08d-0f66d6da4489 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:29.842907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 40/89] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:29.842907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c898e2e-53d5-4462-8b52-ccb686dfe90c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:29.870481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:29.901376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:29.961712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01c0e99-dab1-410c-b08d-0f66d6da4489 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:29.970316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c898e2e-53d5-4462-8b52-ccb686dfe90c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:30.001477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bf59c5-732d-4f6a-9914-5874a37da6b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:30.002388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bf59c5-732d-4f6a-9914-5874a37da6b2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:30.002682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bf59c5-732d-4f6a-9914-5874a37da6b2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:30.002972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bf59c5-732d-4f6a-9914-5874a37da6b2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:30.013547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c974356b-1ab1-4bef-b49e-29437aa48ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:30.014206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c974356b-1ab1-4bef-b49e-29437aa48ec2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:30.014497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c974356b-1ab1-4bef-b49e-29437aa48ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:30.015285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c974356b-1ab1-4bef-b49e-29437aa48ec2 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:30.016932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c974356b-1ab1-4bef-b49e-29437aa48ec2 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:30.018458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c974356b-1ab1-4bef-b49e-29437aa48ec2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:30.020697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 41/90] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:29 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:30.021349 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c974356b-1ab1-4bef-b49e-29437aa48ec2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:30.092438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bf59c5-732d-4f6a-9914-5874a37da6b2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:30.094001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bf59c5-732d-4f6a-9914-5874a37da6b2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:30.138648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:30.151078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:30.154012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c974356b-1ab1-4bef-b49e-29437aa48ec2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:30.202656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05172e9-7a30-4790-b6c3-254fda923ee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:30.203295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05172e9-7a30-4790-b6c3-254fda923ee6 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:30.203590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05172e9-7a30-4790-b6c3-254fda923ee6 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:30.203874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05172e9-7a30-4790-b6c3-254fda923ee6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:30.204978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05172e9-7a30-4790-b6c3-254fda923ee6 None admin] RBAC: Adding query filter params (role.id=9e77a799dd224c1480f51a54f3aed529, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d, user.id=0c530b748ab741c89390231f1b4f34c9) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:30.205301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05172e9-7a30-4790-b6c3-254fda923ee6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:30.207553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05172e9-7a30-4790-b6c3-254fda923ee6 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:30.213015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 50/91] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:30.214091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bf59c5-732d-4f6a-9914-5874a37da6b2 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:30.214842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 42/92] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:30 2022] GET /identity/v3/role_assignments?user.id=0c530b748ab741c89390231f1b4f34c9&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 259 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:30.215672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05172e9-7a30-4790-b6c3-254fda923ee6 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:30.355524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bf59c5-732d-4f6a-9914-5874a37da6b2 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:30.366580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05172e9-7a30-4790-b6c3-254fda923ee6 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:30.414113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17cf5ed-bde4-4116-9b0d-d03db590c847 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:30.414681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17cf5ed-bde4-4116-9b0d-d03db590c847 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:30.414977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17cf5ed-bde4-4116-9b0d-d03db590c847 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:30.415274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17cf5ed-bde4-4116-9b0d-d03db590c847 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:30.416098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17cf5ed-bde4-4116-9b0d-d03db590c847 None admin] RBAC: Adding query filter params (role.id=28ddf4e7cb7f4139a3b6a79077418921, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d, user.id=0c530b748ab741c89390231f1b4f34c9) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:30.416425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17cf5ed-bde4-4116-9b0d-d03db590c847 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:30.420182 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17cf5ed-bde4-4116-9b0d-d03db590c847 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:30.428413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 43/93] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:30 2022] GET /identity/v3/role_assignments?user.id=0c530b748ab741c89390231f1b4f34c9&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=28ddf4e7cb7f4139a3b6a79077418921 => generated 259 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:30.429396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17cf5ed-bde4-4116-9b0d-d03db590c847 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:30.438302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d80260-f567-4769-8edb-cb40a6cbe16c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:30.438994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d80260-f567-4769-8edb-cb40a6cbe16c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:30.439289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d80260-f567-4769-8edb-cb40a6cbe16c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:30.439577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d80260-f567-4769-8edb-cb40a6cbe16c None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:30.445584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d80260-f567-4769-8edb-cb40a6cbe16c None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:30.447146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d80260-f567-4769-8edb-cb40a6cbe16c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:30.449040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 51/94] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:30 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:30.455297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d80260-f567-4769-8edb-cb40a6cbe16c None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:30.469011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:30.497714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:30.679608 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d80260-f567-4769-8edb-cb40a6cbe16c None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:30.686533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17cf5ed-bde4-4116-9b0d-d03db590c847 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:30.720241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc14124-9ba4-498a-beec-45339e9778bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:30.720906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc14124-9ba4-498a-beec-45339e9778bf None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:30.721200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc14124-9ba4-498a-beec-45339e9778bf None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:30.721915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc14124-9ba4-498a-beec-45339e9778bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:30.723269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc14124-9ba4-498a-beec-45339e9778bf None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d, user.id=cfc9fe47ed4740a0aaddec545dbaca1a) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:30.723672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc14124-9ba4-498a-beec-45339e9778bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:30.725781 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc14124-9ba4-498a-beec-45339e9778bf None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:30.731972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 52/95] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:30 2022] GET /identity/v3/role_assignments?user.id=cfc9fe47ed4740a0aaddec545dbaca1a&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 259 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:30.763529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7056f77e-61f1-437c-b2d6-a67eb60beb95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:30.764223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7056f77e-61f1-437c-b2d6-a67eb60beb95 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:30.764520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7056f77e-61f1-437c-b2d6-a67eb60beb95 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:30.764840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7056f77e-61f1-437c-b2d6-a67eb60beb95 None admin] PATH_INFO: `/v3/projects/69ad7185f26745f4bfa315f5ad3baebf` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:30.769684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7056f77e-61f1-437c-b2d6-a67eb60beb95 None admin] RBAC: Authorizing `identity:get_project(project_id=69ad7185f26745f4bfa315f5ad3baebf)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:30.771231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7056f77e-61f1-437c-b2d6-a67eb60beb95 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:30.773300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 44/96] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:30 2022] GET /identity/v3/projects/69ad7185f26745f4bfa315f5ad3baebf => generated 312 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:30.779544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc14124-9ba4-498a-beec-45339e9778bf None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:30.899465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:30.922372 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc14124-9ba4-498a-beec-45339e9778bf None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:30.974860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a02b31-37d6-4539-b234-111e5eb6d863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:30.975478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a02b31-37d6-4539-b234-111e5eb6d863 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:30.975773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a02b31-37d6-4539-b234-111e5eb6d863 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:30.976057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a02b31-37d6-4539-b234-111e5eb6d863 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:30.976902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a02b31-37d6-4539-b234-111e5eb6d863 None admin] RBAC: Adding query filter params (role.id=9e77a799dd224c1480f51a54f3aed529, scope.project.id=69ad7185f26745f4bfa315f5ad3baebf, user.id=0c530b748ab741c89390231f1b4f34c9) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:30.977225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a02b31-37d6-4539-b234-111e5eb6d863 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:30.978739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a02b31-37d6-4539-b234-111e5eb6d863 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:30.987200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 53/97] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:30 2022] GET /identity/v3/role_assignments?user.id=0c530b748ab741c89390231f1b4f34c9&scope.project.id=69ad7185f26745f4bfa315f5ad3baebf&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 259 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:32.450608 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ad5e08-20f6-41fb-ac1a-5d75359f4c86 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:32.451129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ad5e08-20f6-41fb-ac1a-5d75359f4c86 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:32.451204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ad5e08-20f6-41fb-ac1a-5d75359f4c86 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:32.452706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 45/98] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:32.458151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009c359b-f85b-4636-b8a1-a3ff4bcfd6d1 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:32.458575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009c359b-f85b-4636-b8a1-a3ff4bcfd6d1 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:32.458737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009c359b-f85b-4636-b8a1-a3ff4bcfd6d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:32.485899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-009c359b-f85b-4636-b8a1-a3ff4bcfd6d1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:32.487130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-009c359b-f85b-4636-b8a1-a3ff4bcfd6d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:32.505746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7badf8d-d786-4f6e-a2c5-fbe48b813b4c None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:32.506385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7badf8d-d786-4f6e-a2c5-fbe48b813b4c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:32.506869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7badf8d-d786-4f6e-a2c5-fbe48b813b4c None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:32.508439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 46/99] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:32.514706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6706caa5-a610-4484-b83f-82e84913cb91 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:32.515415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6706caa5-a610-4484-b83f-82e84913cb91 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:32.516020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6706caa5-a610-4484-b83f-82e84913cb91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:32.547134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-6706caa5-a610-4484-b83f-82e84913cb91 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:32.548646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-6706caa5-a610-4484-b83f-82e84913cb91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:32.609557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 54/100] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:32.651908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 47/101] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:32.663206 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009c359b-f85b-4636-b8a1-a3ff4bcfd6d1 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:32.690131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6706caa5-a610-4484-b83f-82e84913cb91 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:32.714733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:32.816982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:32.857115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6706caa5-a610-4484-b83f-82e84913cb91 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:32.868396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009c359b-f85b-4636-b8a1-a3ff4bcfd6d1 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:32.898873 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883bf202-eb3f-43e0-ae04-c15d83208cce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:32.899537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883bf202-eb3f-43e0-ae04-c15d83208cce None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:32.899825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883bf202-eb3f-43e0-ae04-c15d83208cce None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:32.900105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883bf202-eb3f-43e0-ae04-c15d83208cce None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:32.920771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883bf202-eb3f-43e0-ae04-c15d83208cce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-883bf202-eb3f-43e0-ae04-c15d83208cce None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:32.926679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:32.927506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:32.928315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 48/102] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:32 2022] GET /identity/v3/roles/reader => generated 84 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:33:32.939406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883bf202-eb3f-43e0-ae04-c15d83208cce None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:32.940381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3747b9-5fd3-4318-9ee3-64a7db967653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:32.941685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3747b9-5fd3-4318-9ee3-64a7db967653 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:32.941967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3747b9-5fd3-4318-9ee3-64a7db967653 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:32.942399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3747b9-5fd3-4318-9ee3-64a7db967653 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:32.943911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3747b9-5fd3-4318-9ee3-64a7db967653 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:32.945404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3747b9-5fd3-4318-9ee3-64a7db967653 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:32.947002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 55/103] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:32.953506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3747b9-5fd3-4318-9ee3-64a7db967653 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:33.108227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:33.123446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3747b9-5fd3-4318-9ee3-64a7db967653 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:33.176285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883bf202-eb3f-43e0-ae04-c15d83208cce None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:33.184330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8af1ce6-b588-4a97-b66d-81ec3636632b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:33.184925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8af1ce6-b588-4a97-b66d-81ec3636632b None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.185222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8af1ce6-b588-4a97-b66d-81ec3636632b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.185510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8af1ce6-b588-4a97-b66d-81ec3636632b None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.186232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8af1ce6-b588-4a97-b66d-81ec3636632b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:33.187711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8af1ce6-b588-4a97-b66d-81ec3636632b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:33.213030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:33.241471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 56/104] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:33:32 2022] POST /identity/v3/projects => generated 302 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:33.247499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cbde8f2-5af2-4a82-b1c1-2e48d13c43a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:33.248200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbde8f2-5af2-4a82-b1c1-2e48d13c43a3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.248493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbde8f2-5af2-4a82-b1c1-2e48d13c43a3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.248803 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cbde8f2-5af2-4a82-b1c1-2e48d13c43a3 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.249730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbde8f2-5af2-4a82-b1c1-2e48d13c43a3 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:33.250116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbde8f2-5af2-4a82-b1c1-2e48d13c43a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:33.252297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cbde8f2-5af2-4a82-b1c1-2e48d13c43a3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:33.268693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 49/105] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:33.277801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8af1ce6-b588-4a97-b66d-81ec3636632b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:33.381057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133db12b-89a2-48c4-892d-51eeef603918 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.381343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133db12b-89a2-48c4-892d-51eeef603918 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.381620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133db12b-89a2-48c4-892d-51eeef603918 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.383306 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 50/106] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:33.395198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ab76de-3fda-4052-982b-4b6bad8e48a2 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.395552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ab76de-3fda-4052-982b-4b6bad8e48a2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.395842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ab76de-3fda-4052-982b-4b6bad8e48a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.405323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8af1ce6-b588-4a97-b66d-81ec3636632b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:33.435567 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-31ab76de-3fda-4052-982b-4b6bad8e48a2 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:33.436863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-31ab76de-3fda-4052-982b-4b6bad8e48a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:33.450518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6202a3-285a-4ac9-8445-4514179553b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:33.451241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6202a3-285a-4ac9-8445-4514179553b1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.451494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6202a3-285a-4ac9-8445-4514179553b1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.451781 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6202a3-285a-4ac9-8445-4514179553b1 None admin] PATH_INFO: `/v3/users/c514bf23041040d39377c6c9fc195a04` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.475594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6202a3-285a-4ac9-8445-4514179553b1 None admin] RBAC: Authorizing `identity:get_user(user_id=c514bf23041040d39377c6c9fc195a04)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:33.477140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6202a3-285a-4ac9-8445-4514179553b1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:33.479102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 57/107] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:33 2022] GET /identity/v3/users/c514bf23041040d39377c6c9fc195a04 => generated 289 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:33.484726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6202a3-285a-4ac9-8445-4514179553b1 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:33.544989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:33.551912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 51/108] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:33.610723 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6202a3-285a-4ac9-8445-4514179553b1 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:33.626049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ab76de-3fda-4052-982b-4b6bad8e48a2 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:33.649944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efbb064-7967-4c64-ab4d-ba609e18bc6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:33.650517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efbb064-7967-4c64-ab4d-ba609e18bc6a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.650809 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efbb064-7967-4c64-ab4d-ba609e18bc6a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.651094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efbb064-7967-4c64-ab4d-ba609e18bc6a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.716983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:33.725204 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efbb064-7967-4c64-ab4d-ba609e18bc6a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:33.726842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0efbb064-7967-4c64-ab4d-ba609e18bc6a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:33.755371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ab76de-3fda-4052-982b-4b6bad8e48a2 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:33.794154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:33.795007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5860991a-77fc-4117-b2c7-99313b95b0e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:33.795835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5860991a-77fc-4117-b2c7-99313b95b0e5 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.796288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5860991a-77fc-4117-b2c7-99313b95b0e5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.796739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5860991a-77fc-4117-b2c7-99313b95b0e5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.814055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5860991a-77fc-4117-b2c7-99313b95b0e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-5860991a-77fc-4117-b2c7-99313b95b0e5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:33.820792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:33.821821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:33.822744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:33.826259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 52/109] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:33.827624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba0a34-e8d7-4653-bec5-8c03078ff28c None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.827922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba0a34-e8d7-4653-bec5-8c03078ff28c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.828209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba0a34-e8d7-4653-bec5-8c03078ff28c None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.829502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 53/110] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:33.830536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cf7a93-e393-49ed-9afc-d3b488b2ea61 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.830828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cf7a93-e393-49ed-9afc-d3b488b2ea61 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.831110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cf7a93-e393-49ed-9afc-d3b488b2ea61 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.832543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 54/111] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:33 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:33.837538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cf7a93-e393-49ed-9afc-d3b488b2ea61 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:33.845038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 58/112] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 29 04:33:33.846353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce3d0b5-57ae-4b8f-9ad3-9fd65ae62e60 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.846646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce3d0b5-57ae-4b8f-9ad3-9fd65ae62e60 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.847434 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce3d0b5-57ae-4b8f-9ad3-9fd65ae62e60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.871135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6ce3d0b5-57ae-4b8f-9ad3-9fd65ae62e60 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:33.872735 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6ce3d0b5-57ae-4b8f-9ad3-9fd65ae62e60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:33.955701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cf7a93-e393-49ed-9afc-d3b488b2ea61 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:33.956870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 59/113] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:33.958794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4559915c-52f2-4644-b8d7-32a9428916fe None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.959301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4559915c-52f2-4644-b8d7-32a9428916fe None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.959796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4559915c-52f2-4644-b8d7-32a9428916fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.980550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4559915c-52f2-4644-b8d7-32a9428916fe None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:33.982291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4559915c-52f2-4644-b8d7-32a9428916fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:33.983210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:33.992358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773cd7ee-f7b5-4093-9073-8f800a49093d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:33.993033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773cd7ee-f7b5-4093-9073-8f800a49093d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:33.993325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773cd7ee-f7b5-4093-9073-8f800a49093d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:33.993606 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773cd7ee-f7b5-4093-9073-8f800a49093d None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:33.994339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773cd7ee-f7b5-4093-9073-8f800a49093d None admin] RBAC: Adding query filter params (name=member) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:33.994674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773cd7ee-f7b5-4093-9073-8f800a49093d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:33.996054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773cd7ee-f7b5-4093-9073-8f800a49093d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:34.005900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 55/114] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:34.006839 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773cd7ee-f7b5-4093-9073-8f800a49093d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:34.065265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 60/115] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:34.073941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4559915c-52f2-4644-b8d7-32a9428916fe None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:34.095537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:34.111235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773cd7ee-f7b5-4093-9073-8f800a49093d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:34.162620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c45ba0-1509-431a-a8f3-af5554d939cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:34.163908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c45ba0-1509-431a-a8f3-af5554d939cc None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.164676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c45ba0-1509-431a-a8f3-af5554d939cc None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.165607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c45ba0-1509-431a-a8f3-af5554d939cc None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.167323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c45ba0-1509-431a-a8f3-af5554d939cc None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:34.169912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c45ba0-1509-431a-a8f3-af5554d939cc None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:34.172070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 56/116] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:34 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:34.173481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c45ba0-1509-431a-a8f3-af5554d939cc None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:34.225870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4559915c-52f2-4644-b8d7-32a9428916fe None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:34.250259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:34.271139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479b2523-abc6-4606-bf06-84b2a0383ed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:34.271853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479b2523-abc6-4606-bf06-84b2a0383ed7 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.272149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479b2523-abc6-4606-bf06-84b2a0383ed7 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.272487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479b2523-abc6-4606-bf06-84b2a0383ed7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.285518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479b2523-abc6-4606-bf06-84b2a0383ed7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-479b2523-abc6-4606-bf06-84b2a0383ed7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:34.291021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:34.291852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:34.293130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 61/117] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:33:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:34.293974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479b2523-abc6-4606-bf06-84b2a0383ed7 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:34.307068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c45ba0-1509-431a-a8f3-af5554d939cc None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:34.347040 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1091aa6-4c68-448c-af98-3ab25559ce02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:34.348295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1091aa6-4c68-448c-af98-3ab25559ce02 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.348588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1091aa6-4c68-448c-af98-3ab25559ce02 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.348908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1091aa6-4c68-448c-af98-3ab25559ce02 None admin] PATH_INFO: `/v3/users/0c530b748ab741c89390231f1b4f34c9` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.350121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1091aa6-4c68-448c-af98-3ab25559ce02 None admin] RBAC: Authorizing `identity:get_user(user_id=0c530b748ab741c89390231f1b4f34c9)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:34.351583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1091aa6-4c68-448c-af98-3ab25559ce02 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:34.353253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 57/118] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:34 2022] GET /identity/v3/users/0c530b748ab741c89390231f1b4f34c9 => generated 275 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:34.354068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1091aa6-4c68-448c-af98-3ab25559ce02 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:34.383985 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:34.416259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479b2523-abc6-4606-bf06-84b2a0383ed7 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:34.459290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7fb93e-1dc0-4bcf-a0cf-67694ad9ca96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:34.459944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7fb93e-1dc0-4bcf-a0cf-67694ad9ca96 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.460241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7fb93e-1dc0-4bcf-a0cf-67694ad9ca96 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.460522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7fb93e-1dc0-4bcf-a0cf-67694ad9ca96 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.461451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1091aa6-4c68-448c-af98-3ab25559ce02 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:34.471280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7fb93e-1dc0-4bcf-a0cf-67694ad9ca96 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-9c7fb93e-1dc0-4bcf-a0cf-67694ad9ca96 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:34.476624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:34.477872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 62/119] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:33:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:34.478603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7fb93e-1dc0-4bcf-a0cf-67694ad9ca96 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:34.492638 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:34.502595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059087e0-2a79-4ffd-b6d4-5f94a244aad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:34.503756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059087e0-2a79-4ffd-b6d4-5f94a244aad9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.504173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059087e0-2a79-4ffd-b6d4-5f94a244aad9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.504571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059087e0-2a79-4ffd-b6d4-5f94a244aad9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.506306 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059087e0-2a79-4ffd-b6d4-5f94a244aad9 None admin] RBAC: Adding query filter params (role.id=f382f86b952e4b869017bbf06295b028, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d, user.id=c514bf23041040d39377c6c9fc195a04) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:34.506784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059087e0-2a79-4ffd-b6d4-5f94a244aad9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:34.508253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059087e0-2a79-4ffd-b6d4-5f94a244aad9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:34.514111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 58/120] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:34 2022] GET /identity/v3/role_assignments?user.id=c514bf23041040d39377c6c9fc195a04&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=f382f86b952e4b869017bbf06295b028 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:34.522364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb23bef-d6b4-4f31-b6c3-386d2ce9ba2d None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.522364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb23bef-d6b4-4f31-b6c3-386d2ce9ba2d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.522364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb23bef-d6b4-4f31-b6c3-386d2ce9ba2d None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.522580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 59/121] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:34.523383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb23bef-d6b4-4f31-b6c3-386d2ce9ba2d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:34.603652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7fb93e-1dc0-4bcf-a0cf-67694ad9ca96 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:34.641576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf14ad2c-47ba-4eb7-8ef2-db794543f51e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:34.642212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf14ad2c-47ba-4eb7-8ef2-db794543f51e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.642498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf14ad2c-47ba-4eb7-8ef2-db794543f51e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.642776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf14ad2c-47ba-4eb7-8ef2-db794543f51e None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.643640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf14ad2c-47ba-4eb7-8ef2-db794543f51e None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:34.644229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf14ad2c-47ba-4eb7-8ef2-db794543f51e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:34.645368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf14ad2c-47ba-4eb7-8ef2-db794543f51e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:34.646977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:34.670179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 63/122] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:34.671251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf14ad2c-47ba-4eb7-8ef2-db794543f51e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:34.680761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb23bef-d6b4-4f31-b6c3-386d2ce9ba2d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:34.719862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029fa081-ade3-4f7e-b753-1b335324c151 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:34.720852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029fa081-ade3-4f7e-b753-1b335324c151 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.721325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029fa081-ade3-4f7e-b753-1b335324c151 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.721816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029fa081-ade3-4f7e-b753-1b335324c151 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.723286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029fa081-ade3-4f7e-b753-1b335324c151 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:34.725063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029fa081-ade3-4f7e-b753-1b335324c151 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:34.727013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 60/123] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:34 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:34.728535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683ada0-18df-4d57-a574-45a3c4c7aca5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.729039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683ada0-18df-4d57-a574-45a3c4c7aca5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.729506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683ada0-18df-4d57-a574-45a3c4c7aca5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.748587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-4683ada0-18df-4d57-a574-45a3c4c7aca5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:34.750152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-4683ada0-18df-4d57-a574-45a3c4c7aca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:34.800834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf14ad2c-47ba-4eb7-8ef2-db794543f51e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:34.806738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:34.830461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 61/124] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:34.831374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4683ada0-18df-4d57-a574-45a3c4c7aca5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:34.849580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14343b8-f902-4b7c-beb1-46fcd4001299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:34.850544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14343b8-f902-4b7c-beb1-46fcd4001299 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.851033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14343b8-f902-4b7c-beb1-46fcd4001299 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.851513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14343b8-f902-4b7c-beb1-46fcd4001299 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.852462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14343b8-f902-4b7c-beb1-46fcd4001299 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:34.853026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14343b8-f902-4b7c-beb1-46fcd4001299 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:34.854636 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14343b8-f902-4b7c-beb1-46fcd4001299 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:34.864292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 64/125] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:33:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:34.868918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14343b8-f902-4b7c-beb1-46fcd4001299 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:34.932982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:34.937015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4683ada0-18df-4d57-a574-45a3c4c7aca5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:34.973568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9650bdb-77d9-49ac-8c32-854e66afb82c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:34.974245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9650bdb-77d9-49ac-8c32-854e66afb82c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:34.974513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9650bdb-77d9-49ac-8c32-854e66afb82c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:34.974807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9650bdb-77d9-49ac-8c32-854e66afb82c None admin] PATH_INFO: `/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:34.975992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9650bdb-77d9-49ac-8c32-854e66afb82c None admin] RBAC: Authorizing `identity:get_user(user_id=cfc9fe47ed4740a0aaddec545dbaca1a)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:34.977264 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14343b8-f902-4b7c-beb1-46fcd4001299 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:34.977480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9650bdb-77d9-49ac-8c32-854e66afb82c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:34.979361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 62/126] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:34 2022] GET /identity/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a => generated 247 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:34.979894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9650bdb-77d9-49ac-8c32-854e66afb82c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:35.015341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafbfe8a-2009-4320-aefd-4733791a633b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:35.016466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafbfe8a-2009-4320-aefd-4733791a633b None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:35.017013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafbfe8a-2009-4320-aefd-4733791a633b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:35.017495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafbfe8a-2009-4320-aefd-4733791a633b None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/users/0c530b748ab741c89390231f1b4f34c9/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:35.019880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafbfe8a-2009-4320-aefd-4733791a633b None admin] RBAC: Authorizing `identity:create_grant(project_id=e96107c2d25740e783b49bd3a96cbc8d, user_id=0c530b748ab741c89390231f1b4f34c9, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:35.021591 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafbfe8a-2009-4320-aefd-4733791a633b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:35.039113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 65/127] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:33:34 2022] PUT /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/users/0c530b748ab741c89390231f1b4f34c9/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:33:35.040702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafbfe8a-2009-4320-aefd-4733791a633b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:35.041259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:35.096177 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9650bdb-77d9-49ac-8c32-854e66afb82c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:35.139477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f83417-08ec-482f-a485-e28309386d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:35.140134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f83417-08ec-482f-a485-e28309386d62 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:35.140463 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f83417-08ec-482f-a485-e28309386d62 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:35.140777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f83417-08ec-482f-a485-e28309386d62 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:35.155160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f83417-08ec-482f-a485-e28309386d62 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-95f83417-08ec-482f-a485-e28309386d62 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:35.160766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:35.161764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 63/128] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:35.163000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f83417-08ec-482f-a485-e28309386d62 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:35.184189 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafbfe8a-2009-4320-aefd-4733791a633b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:35.206176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:35.232719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2b342-b781-40bb-9d68-680be1afe5c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:35.234406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2b342-b781-40bb-9d68-680be1afe5c2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:35.234919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2b342-b781-40bb-9d68-680be1afe5c2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:35.235412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2b342-b781-40bb-9d68-680be1afe5c2 None admin] PATH_INFO: `/v3/users/0c530b748ab741c89390231f1b4f34c9` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:35.237415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e2b342-b781-40bb-9d68-680be1afe5c2 None admin] RBAC: Authorizing `identity:get_user(user_id=0c530b748ab741c89390231f1b4f34c9)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:35.240050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e2b342-b781-40bb-9d68-680be1afe5c2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:35.242419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 66/129] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:35 2022] GET /identity/v3/users/0c530b748ab741c89390231f1b4f34c9 => generated 275 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:35.243637 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2b342-b781-40bb-9d68-680be1afe5c2 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:35.317982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f83417-08ec-482f-a485-e28309386d62 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:35.336996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:35.356125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd226c6-47a3-4f6b-a4c3-0bb768e01c5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:35.357116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd226c6-47a3-4f6b-a4c3-0bb768e01c5a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:35.357243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd226c6-47a3-4f6b-a4c3-0bb768e01c5a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:35.357567 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd226c6-47a3-4f6b-a4c3-0bb768e01c5a None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:35.358778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd226c6-47a3-4f6b-a4c3-0bb768e01c5a None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:35.360267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd226c6-47a3-4f6b-a4c3-0bb768e01c5a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:35.360849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2b342-b781-40bb-9d68-680be1afe5c2 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:35.362076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 64/130] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:35 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:35.363107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd226c6-47a3-4f6b-a4c3-0bb768e01c5a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:35.412106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d300497-ee9a-445e-95cc-20df0b0319d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:35.412782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d300497-ee9a-445e-95cc-20df0b0319d9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:35.413073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d300497-ee9a-445e-95cc-20df0b0319d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:35.413352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d300497-ee9a-445e-95cc-20df0b0319d9 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:35.414090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d300497-ee9a-445e-95cc-20df0b0319d9 None admin] RBAC: Adding query filter params (name=member) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:35.414399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d300497-ee9a-445e-95cc-20df0b0319d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:35.415808 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d300497-ee9a-445e-95cc-20df0b0319d9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:35.446867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 67/131] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:35.447725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d300497-ee9a-445e-95cc-20df0b0319d9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:35.472667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd226c6-47a3-4f6b-a4c3-0bb768e01c5a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:35.514832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a5aa9b-010f-49c9-a403-12405c02adb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:35.515406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a5aa9b-010f-49c9-a403-12405c02adb5 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:35.515735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a5aa9b-010f-49c9-a403-12405c02adb5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:35.516026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a5aa9b-010f-49c9-a403-12405c02adb5 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:35.519721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a5aa9b-010f-49c9-a403-12405c02adb5 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:35.521191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a5aa9b-010f-49c9-a403-12405c02adb5 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:35.524334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 65/132] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:35 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:35.525146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a5aa9b-010f-49c9-a403-12405c02adb5 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:35.561685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:35.628030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:35.660016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d300497-ee9a-445e-95cc-20df0b0319d9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:35.717995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2591024c-6493-4c57-a7d3-740721611bba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:35.718739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2591024c-6493-4c57-a7d3-740721611bba None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:35.719032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2591024c-6493-4c57-a7d3-740721611bba None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:35.719314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2591024c-6493-4c57-a7d3-740721611bba None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/users/cfc9fe47ed4740a0aaddec545dbaca1a/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:35.721328 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2591024c-6493-4c57-a7d3-740721611bba None admin] RBAC: Authorizing `identity:create_grant(project_id=e96107c2d25740e783b49bd3a96cbc8d, user_id=cfc9fe47ed4740a0aaddec545dbaca1a, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:35.722868 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2591024c-6493-4c57-a7d3-740721611bba None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:35.723401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a5aa9b-010f-49c9-a403-12405c02adb5 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:35.742950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 68/133] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:33:35 2022] PUT /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/users/cfc9fe47ed4740a0aaddec545dbaca1a/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:33:35.743538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2591024c-6493-4c57-a7d3-740721611bba None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:35.803044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c71ac2-1b34-42dd-8b27-cedc4490505d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:35.803666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c71ac2-1b34-42dd-8b27-cedc4490505d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:35.803961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c71ac2-1b34-42dd-8b27-cedc4490505d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:35.804239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c71ac2-1b34-42dd-8b27-cedc4490505d None admin] PATH_INFO: `/v3/users/0c530b748ab741c89390231f1b4f34c9` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:35.805605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c71ac2-1b34-42dd-8b27-cedc4490505d None admin] RBAC: Authorizing `identity:get_user(user_id=0c530b748ab741c89390231f1b4f34c9)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:35.807337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c71ac2-1b34-42dd-8b27-cedc4490505d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:35.818095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 66/134] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:35 2022] GET /identity/v3/users/0c530b748ab741c89390231f1b4f34c9 => generated 275 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:35.823567 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c71ac2-1b34-42dd-8b27-cedc4490505d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:35.972382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2591024c-6493-4c57-a7d3-740721611bba None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:35.984093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:36.020510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181164e2-175d-41e2-8fb1-0ea4378e5e57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:36.021495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181164e2-175d-41e2-8fb1-0ea4378e5e57 None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:36.022084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181164e2-175d-41e2-8fb1-0ea4378e5e57 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:36.022889 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181164e2-175d-41e2-8fb1-0ea4378e5e57 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/users/0c530b748ab741c89390231f1b4f34c9/roles/28ddf4e7cb7f4139a3b6a79077418921` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:36.025855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181164e2-175d-41e2-8fb1-0ea4378e5e57 None admin] RBAC: Authorizing `identity:create_grant(project_id=e96107c2d25740e783b49bd3a96cbc8d, user_id=0c530b748ab741c89390231f1b4f34c9, role_id=28ddf4e7cb7f4139a3b6a79077418921)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:36.028141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181164e2-175d-41e2-8fb1-0ea4378e5e57 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:36.034327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c71ac2-1b34-42dd-8b27-cedc4490505d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:36.044846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 69/135] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:33:35 2022] PUT /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/users/0c530b748ab741c89390231f1b4f34c9/roles/28ddf4e7cb7f4139a3b6a79077418921 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:33:36.046420 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240692d-b3cf-4d22-899b-a701c074b1b5 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:36.046867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240692d-b3cf-4d22-899b-a701c074b1b5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:36.047306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9240692d-b3cf-4d22-899b-a701c074b1b5 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:36.049330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 70/136] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:36.056507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a442f-7faa-40b5-bf9b-2f3e37cd87b6 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:36.058719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a442f-7faa-40b5-bf9b-2f3e37cd87b6 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:36.059238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a442f-7faa-40b5-bf9b-2f3e37cd87b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:36.074907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:36.091100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c14a442f-7faa-40b5-bf9b-2f3e37cd87b6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:36.092884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c14a442f-7faa-40b5-bf9b-2f3e37cd87b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:36.117925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c563c65a-8fe5-4ba9-9ea4-b9f5ddce7977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:36.118630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563c65a-8fe5-4ba9-9ea4-b9f5ddce7977 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:36.118918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563c65a-8fe5-4ba9-9ea4-b9f5ddce7977 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:36.119370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563c65a-8fe5-4ba9-9ea4-b9f5ddce7977 None admin] PATH_INFO: `/v3/projects/69ad7185f26745f4bfa315f5ad3baebf` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:36.125331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c563c65a-8fe5-4ba9-9ea4-b9f5ddce7977 None admin] RBAC: Authorizing `identity:get_project(project_id=69ad7185f26745f4bfa315f5ad3baebf)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:36.126805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c563c65a-8fe5-4ba9-9ea4-b9f5ddce7977 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:36.129626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 67/137] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:35 2022] GET /identity/v3/projects/69ad7185f26745f4bfa315f5ad3baebf => generated 312 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:36.137738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c563c65a-8fe5-4ba9-9ea4-b9f5ddce7977 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:36.186757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 71/138] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:36.212374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14a442f-7faa-40b5-bf9b-2f3e37cd87b6 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:36.332877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14a442f-7faa-40b5-bf9b-2f3e37cd87b6 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:36.344282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c563c65a-8fe5-4ba9-9ea4-b9f5ddce7977 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:36.353023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:36.403440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cdb13d-c980-4176-8719-40afe584e46d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:36.404209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cdb13d-c980-4176-8719-40afe584e46d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:36.404499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cdb13d-c980-4176-8719-40afe584e46d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:36.404891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cdb13d-c980-4176-8719-40afe584e46d None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:36.406233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cdb13d-c980-4176-8719-40afe584e46d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:36.406703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16eaff94-7ab6-4ca4-af43-203a8390db29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:36.407557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eaff94-7ab6-4ca4-af43-203a8390db29 None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:36.407667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cdb13d-c980-4176-8719-40afe584e46d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:36.408083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eaff94-7ab6-4ca4-af43-203a8390db29 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:36.408513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eaff94-7ab6-4ca4-af43-203a8390db29 None admin] PATH_INFO: `/v3/projects/69ad7185f26745f4bfa315f5ad3baebf/users/0c530b748ab741c89390231f1b4f34c9/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:36.409308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 72/139] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:36.415620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cdb13d-c980-4176-8719-40afe584e46d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:36.416074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16eaff94-7ab6-4ca4-af43-203a8390db29 None admin] RBAC: Authorizing `identity:create_grant(project_id=69ad7185f26745f4bfa315f5ad3baebf, user_id=0c530b748ab741c89390231f1b4f34c9, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:36.417832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16eaff94-7ab6-4ca4-af43-203a8390db29 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:36.436878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 68/140] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:33:36 2022] PUT /identity/v3/projects/69ad7185f26745f4bfa315f5ad3baebf/users/0c530b748ab741c89390231f1b4f34c9/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:33:36.571945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cdb13d-c980-4176-8719-40afe584e46d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:36.620077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036fdae0-8e6a-48e7-9877-76a60324e07e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:36.620644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036fdae0-8e6a-48e7-9877-76a60324e07e None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:36.620943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036fdae0-8e6a-48e7-9877-76a60324e07e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:36.621226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036fdae0-8e6a-48e7-9877-76a60324e07e None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:36.621938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036fdae0-8e6a-48e7-9877-76a60324e07e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:36.623410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036fdae0-8e6a-48e7-9877-76a60324e07e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:36.667037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 73/141] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:33:36 2022] POST /identity/v3/users => generated 283 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:33:37.207195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b29895-a6ba-4139-84a4-26341da155a4 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:37.208205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b29895-a6ba-4139-84a4-26341da155a4 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:37.208757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93b29895-a6ba-4139-84a4-26341da155a4 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:37.210377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 69/142] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:37.215969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d8212a-c9f8-4bcf-a963-04b6de0b8862 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:37.216269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d8212a-c9f8-4bcf-a963-04b6de0b8862 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:37.216550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d8212a-c9f8-4bcf-a963-04b6de0b8862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:37.242890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-01d8212a-c9f8-4bcf-a963-04b6de0b8862 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:37.244107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-01d8212a-c9f8-4bcf-a963-04b6de0b8862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:37.268318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:37.330493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 74/143] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:37.356343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b29895-a6ba-4139-84a4-26341da155a4 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:37.445646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:37.529488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93b29895-a6ba-4139-84a4-26341da155a4 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:37.537851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570a5ae5-bdcf-4577-a94f-12a2324a43b6 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:37.538148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570a5ae5-bdcf-4577-a94f-12a2324a43b6 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:37.538424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570a5ae5-bdcf-4577-a94f-12a2324a43b6 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:37.540056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 75/144] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:37.545156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976be470-f118-4dcd-b7c7-106df4b434cb None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:37.545450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976be470-f118-4dcd-b7c7-106df4b434cb None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:37.545737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976be470-f118-4dcd-b7c7-106df4b434cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:37.574786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a81a20b-7e0c-40f9-8a82-d6f2c6991dbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:37.575460 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a81a20b-7e0c-40f9-8a82-d6f2c6991dbd None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:37.575845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a81a20b-7e0c-40f9-8a82-d6f2c6991dbd None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:37.576141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a81a20b-7e0c-40f9-8a82-d6f2c6991dbd None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:37.582192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-976be470-f118-4dcd-b7c7-106df4b434cb None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:37.583505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-976be470-f118-4dcd-b7c7-106df4b434cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:37.593517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a81a20b-7e0c-40f9-8a82-d6f2c6991dbd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-9a81a20b-7e0c-40f9-8a82-d6f2c6991dbd None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:37.598670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:37.599697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:37.600717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 70/145] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:37.606538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a81a20b-7e0c-40f9-8a82-d6f2c6991dbd None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:37.692153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 76/146] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:37.755921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976be470-f118-4dcd-b7c7-106df4b434cb None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:37.767908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a81a20b-7e0c-40f9-8a82-d6f2c6991dbd None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:37.783636 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:37.842110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbf4e2e-414b-4dfa-bb57-106ad3ebc8aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:37.842757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbf4e2e-414b-4dfa-bb57-106ad3ebc8aa None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:37.843056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbf4e2e-414b-4dfa-bb57-106ad3ebc8aa None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:37.843341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbf4e2e-414b-4dfa-bb57-106ad3ebc8aa None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:37.844082 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbf4e2e-414b-4dfa-bb57-106ad3ebc8aa None admin] RBAC: Adding query filter params (name=reader) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:37.844396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbf4e2e-414b-4dfa-bb57-106ad3ebc8aa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:37.845903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efbf4e2e-414b-4dfa-bb57-106ad3ebc8aa None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:37.858208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 71/147] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:37 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:37.865611 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbf4e2e-414b-4dfa-bb57-106ad3ebc8aa None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:37.906743 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:37.961794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976be470-f118-4dcd-b7c7-106df4b434cb None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:38.036479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ff2255-1cc0-445c-935b-8c32fb9245f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:38.037145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ff2255-1cc0-445c-935b-8c32fb9245f8 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:38.037433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ff2255-1cc0-445c-935b-8c32fb9245f8 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:38.037712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ff2255-1cc0-445c-935b-8c32fb9245f8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:38.062724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ff2255-1cc0-445c-935b-8c32fb9245f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-54ff2255-1cc0-445c-935b-8c32fb9245f8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:38.069767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:38.070611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:38.071366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:38.073027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 77/148] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:33:38.078777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ff2255-1cc0-445c-935b-8c32fb9245f8 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:38.085551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbf4e2e-414b-4dfa-bb57-106ad3ebc8aa None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:38.152852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e37f6e-4960-4d4b-9940-c05e8209a810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:38.153522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e37f6e-4960-4d4b-9940-c05e8209a810 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:38.153818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e37f6e-4960-4d4b-9940-c05e8209a810 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:38.154121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e37f6e-4960-4d4b-9940-c05e8209a810 None admin] PATH_INFO: `/v3/users/c514bf23041040d39377c6c9fc195a04` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:38.156133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e37f6e-4960-4d4b-9940-c05e8209a810 None admin] RBAC: Authorizing `identity:get_user(user_id=c514bf23041040d39377c6c9fc195a04)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:38.157618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e37f6e-4960-4d4b-9940-c05e8209a810 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:38.164765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 72/149] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:37 2022] GET /identity/v3/users/c514bf23041040d39377c6c9fc195a04 => generated 289 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:38.171360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e37f6e-4960-4d4b-9940-c05e8209a810 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:38.240810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:38.294400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ff2255-1cc0-445c-935b-8c32fb9245f8 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:38.361376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b941b07-aaa7-4324-89e7-db423ba259af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:38.362176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b941b07-aaa7-4324-89e7-db423ba259af None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:38.362549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b941b07-aaa7-4324-89e7-db423ba259af None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:38.363397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b941b07-aaa7-4324-89e7-db423ba259af None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:38.364146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b941b07-aaa7-4324-89e7-db423ba259af None admin] RBAC: Adding query filter params (name=member) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:38.364454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b941b07-aaa7-4324-89e7-db423ba259af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:38.365667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e37f6e-4960-4d4b-9940-c05e8209a810 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:38.367991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b941b07-aaa7-4324-89e7-db423ba259af None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:38.376934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 78/150] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:38.382684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b941b07-aaa7-4324-89e7-db423ba259af None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:38.405283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:38.430643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ef7a7-3ced-4fbe-b439-da8b74fd36ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:38.431358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ef7a7-3ced-4fbe-b439-da8b74fd36ad None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:38.431650 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ef7a7-3ced-4fbe-b439-da8b74fd36ad None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:38.432012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919ef7a7-3ced-4fbe-b439-da8b74fd36ad None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:38.433782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919ef7a7-3ced-4fbe-b439-da8b74fd36ad None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:38.435489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919ef7a7-3ced-4fbe-b439-da8b74fd36ad None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:38.438554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 73/151] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:38 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:38.446231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ef7a7-3ced-4fbe-b439-da8b74fd36ad None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:38.549635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b941b07-aaa7-4324-89e7-db423ba259af None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:38.618339 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef497f-d163-4124-beb3-7f1527647ae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:38.619108 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef497f-d163-4124-beb3-7f1527647ae3 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:38.619288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef497f-d163-4124-beb3-7f1527647ae3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:38.619567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ef497f-d163-4124-beb3-7f1527647ae3 None admin] PATH_INFO: `/v3/users/0c530b748ab741c89390231f1b4f34c9` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:38.625395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ef497f-d163-4124-beb3-7f1527647ae3 None admin] RBAC: Authorizing `identity:get_user(user_id=0c530b748ab741c89390231f1b4f34c9)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:38.626885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ef497f-d163-4124-beb3-7f1527647ae3 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:38.629013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 79/152] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:38 2022] GET /identity/v3/users/0c530b748ab741c89390231f1b4f34c9 => generated 275 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:38.635918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef497f-d163-4124-beb3-7f1527647ae3 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:38.691551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919ef7a7-3ced-4fbe-b439-da8b74fd36ad None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:38.703874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:38.750447 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee1c03b-66ea-4337-b8d4-0cc118965023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:38.751145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee1c03b-66ea-4337-b8d4-0cc118965023 None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:38.751391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:38.751490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee1c03b-66ea-4337-b8d4-0cc118965023 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:38.751716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee1c03b-66ea-4337-b8d4-0cc118965023 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/users/c514bf23041040d39377c6c9fc195a04/roles/f382f86b952e4b869017bbf06295b028` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:38.754017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee1c03b-66ea-4337-b8d4-0cc118965023 None admin] RBAC: Authorizing `identity:create_grant(project_id=e96107c2d25740e783b49bd3a96cbc8d, user_id=c514bf23041040d39377c6c9fc195a04, role_id=f382f86b952e4b869017bbf06295b028)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:38.755524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee1c03b-66ea-4337-b8d4-0cc118965023 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:38.771142 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 74/153] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:33:38 2022] PUT /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/users/c514bf23041040d39377c6c9fc195a04/roles/f382f86b952e4b869017bbf06295b028 => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:33:38.813014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ef497f-d163-4124-beb3-7f1527647ae3 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:38.865192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9df36d-4305-4095-a638-629c1ea4c862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:38.865837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9df36d-4305-4095-a638-629c1ea4c862 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:38.866123 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9df36d-4305-4095-a638-629c1ea4c862 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:38.866405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9df36d-4305-4095-a638-629c1ea4c862 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:38.958483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9df36d-4305-4095-a638-629c1ea4c862 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:38.960164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9df36d-4305-4095-a638-629c1ea4c862 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:39.053072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491ba7cd-9b7b-45f7-a2f0-6afd2e53fabc None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:39.053384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491ba7cd-9b7b-45f7-a2f0-6afd2e53fabc None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:39.053652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491ba7cd-9b7b-45f7-a2f0-6afd2e53fabc None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:39.055305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 75/154] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:39.059796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:39.062760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e095ba4-7da8-4297-a13e-2f8b5f11db05 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:39.063122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e095ba4-7da8-4297-a13e-2f8b5f11db05 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:39.063323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e095ba4-7da8-4297-a13e-2f8b5f11db05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:39.102217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-8e095ba4-7da8-4297-a13e-2f8b5f11db05 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:39.103450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8e095ba4-7da8-4297-a13e-2f8b5f11db05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:39.125412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 80/155] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 29 04:33:39.131970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9df36d-4305-4095-a638-629c1ea4c862 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:39.262821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 76/156] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:39.317268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9df36d-4305-4095-a638-629c1ea4c862 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:39.321933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e095ba4-7da8-4297-a13e-2f8b5f11db05 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:39.373368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b3f562-c566-4ebb-8d85-5cf7edb69b03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:39.374280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b3f562-c566-4ebb-8d85-5cf7edb69b03 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:39.374645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b3f562-c566-4ebb-8d85-5cf7edb69b03 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:39.375062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b3f562-c566-4ebb-8d85-5cf7edb69b03 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:39.381624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b3f562-c566-4ebb-8d85-5cf7edb69b03 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:39.383906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b3f562-c566-4ebb-8d85-5cf7edb69b03 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:39.385817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 81/157] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:39 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:39.391883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b3f562-c566-4ebb-8d85-5cf7edb69b03 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:39.460981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:39.525715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:39.540432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e095ba4-7da8-4297-a13e-2f8b5f11db05 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:39.571783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b3f562-c566-4ebb-8d85-5cf7edb69b03 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:39.625891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958faa20-ae11-4e2b-8cc0-f42f37f27db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:39.626594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958faa20-ae11-4e2b-8cc0-f42f37f27db3 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:39.626902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958faa20-ae11-4e2b-8cc0-f42f37f27db3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:39.627079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958faa20-ae11-4e2b-8cc0-f42f37f27db3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:39.627878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958faa20-ae11-4e2b-8cc0-f42f37f27db3 None admin] RBAC: Adding query filter params (role.id=9e77a799dd224c1480f51a54f3aed529, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d, user.id=0c530b748ab741c89390231f1b4f34c9) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:39.628374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958faa20-ae11-4e2b-8cc0-f42f37f27db3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:39.629313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51460e0-ee09-4f26-9bb6-7feec21de72d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:39.629977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51460e0-ee09-4f26-9bb6-7feec21de72d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:39.630265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958faa20-ae11-4e2b-8cc0-f42f37f27db3 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:39.630366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51460e0-ee09-4f26-9bb6-7feec21de72d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:39.630540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51460e0-ee09-4f26-9bb6-7feec21de72d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:39.642062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 82/158] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:39 2022] GET /identity/v3/role_assignments?user.id=0c530b748ab741c89390231f1b4f34c9&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 608 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:39.643362 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c15f8-0c68-448a-bac0-1ea88af0ebe6 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:39.644228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c15f8-0c68-448a-bac0-1ea88af0ebe6 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:39.644652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622c15f8-0c68-448a-bac0-1ea88af0ebe6 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:39.646797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 83/159] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:39.658841 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c387d5-289b-4d18-9260-c36e1ffe5a8c None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:39.659134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c387d5-289b-4d18-9260-c36e1ffe5a8c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:39.659483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c387d5-289b-4d18-9260-c36e1ffe5a8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:39.676970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51460e0-ee09-4f26-9bb6-7feec21de72d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-e51460e0-ee09-4f26-9bb6-7feec21de72d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:39.682012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:39.682850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:39.683626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 77/160] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:33:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 362 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:39.689839 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51460e0-ee09-4f26-9bb6-7feec21de72d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:39.698021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-60c387d5-289b-4d18-9260-c36e1ffe5a8c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:39.699799 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-60c387d5-289b-4d18-9260-c36e1ffe5a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:39.808264 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 84/161] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:39.834120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c387d5-289b-4d18-9260-c36e1ffe5a8c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:39.897078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:39.986791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51460e0-ee09-4f26-9bb6-7feec21de72d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:40.024187 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c387d5-289b-4d18-9260-c36e1ffe5a8c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:40.058023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd27778-9cc5-4d31-bb8c-616832ceae45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:40.058658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd27778-9cc5-4d31-bb8c-616832ceae45 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:40.058948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd27778-9cc5-4d31-bb8c-616832ceae45 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:40.059229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd27778-9cc5-4d31-bb8c-616832ceae45 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:40.059987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd27778-9cc5-4d31-bb8c-616832ceae45 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:40.060370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd27778-9cc5-4d31-bb8c-616832ceae45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:40.062060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd27778-9cc5-4d31-bb8c-616832ceae45 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:40.067055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49b1918-3f7c-4ee8-9b4b-fc245a71e4dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:40.067894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b1918-3f7c-4ee8-9b4b-fc245a71e4dc None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:40.068344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b1918-3f7c-4ee8-9b4b-fc245a71e4dc None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:40.068800 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b1918-3f7c-4ee8-9b4b-fc245a71e4dc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:40.082021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49b1918-3f7c-4ee8-9b4b-fc245a71e4dc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:40.084889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 78/162] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 391 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:40.087432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd27778-9cc5-4d31-bb8c-616832ceae45 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-f49b1918-3f7c-4ee8-9b4b-fc245a71e4dc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:40.088234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:40.089309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:40.090285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:40.092341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 85/163] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:33:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:40.099638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49b1918-3f7c-4ee8-9b4b-fc245a71e4dc None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:40.123983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:40.234927 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:40.241352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49b1918-3f7c-4ee8-9b4b-fc245a71e4dc None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:40.309584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bb66eb-c6f1-472a-810f-0a5f6262d404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:40.310550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bb66eb-c6f1-472a-810f-0a5f6262d404 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:40.311138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bb66eb-c6f1-472a-810f-0a5f6262d404 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:40.311559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bb66eb-c6f1-472a-810f-0a5f6262d404 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:40.312424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bb66eb-c6f1-472a-810f-0a5f6262d404 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:40.312913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bb66eb-c6f1-472a-810f-0a5f6262d404 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:40.315325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bb66eb-c6f1-472a-810f-0a5f6262d404 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:40.332580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 86/164] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:33:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:40.339628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bb66eb-c6f1-472a-810f-0a5f6262d404 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:40.342711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd27778-9cc5-4d31-bb8c-616832ceae45 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:40.415453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7490e7c1-43f1-45a2-9a1f-c556daed766e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:40.416124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7490e7c1-43f1-45a2-9a1f-c556daed766e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:40.416414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7490e7c1-43f1-45a2-9a1f-c556daed766e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:40.416731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7490e7c1-43f1-45a2-9a1f-c556daed766e None admin] PATH_INFO: `/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:40.418316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7490e7c1-43f1-45a2-9a1f-c556daed766e None admin] RBAC: Authorizing `identity:get_user(user_id=cfc9fe47ed4740a0aaddec545dbaca1a)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:40.419842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7490e7c1-43f1-45a2-9a1f-c556daed766e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:40.423932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 79/165] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:40 2022] GET /identity/v3/users/cfc9fe47ed4740a0aaddec545dbaca1a => generated 247 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:40.430729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b608cf8d-d3b3-471e-b6dd-38b98f841359 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:40.431017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b608cf8d-d3b3-471e-b6dd-38b98f841359 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:40.431297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b608cf8d-d3b3-471e-b6dd-38b98f841359 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:40.432696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 80/166] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:40 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:40.433448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b608cf8d-d3b3-471e-b6dd-38b98f841359 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:40.567159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bb66eb-c6f1-472a-810f-0a5f6262d404 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:40.634118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:40.648783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c3e5b9-1e55-4bb2-afb7-0afdd5b868ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:40.650154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3e5b9-1e55-4bb2-afb7-0afdd5b868ad None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:40.650644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3e5b9-1e55-4bb2-afb7-0afdd5b868ad None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:40.651125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c3e5b9-1e55-4bb2-afb7-0afdd5b868ad None admin] PATH_INFO: `/v3/users/0c530b748ab741c89390231f1b4f34c9` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:40.653623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c3e5b9-1e55-4bb2-afb7-0afdd5b868ad None admin] RBAC: Authorizing `identity:get_user(user_id=0c530b748ab741c89390231f1b4f34c9)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:40.656625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c3e5b9-1e55-4bb2-afb7-0afdd5b868ad None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:40.657499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 87/167] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:40 2022] GET /identity/v3/users/0c530b748ab741c89390231f1b4f34c9 => generated 275 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:40.658527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14089046-6833-43c0-b652-6ebdc2325225 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:40.658818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14089046-6833-43c0-b652-6ebdc2325225 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:40.659101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14089046-6833-43c0-b652-6ebdc2325225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:40.702477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-14089046-6833-43c0-b652-6ebdc2325225 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:40.703425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b608cf8d-d3b3-471e-b6dd-38b98f841359 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:40.703682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-14089046-6833-43c0-b652-6ebdc2325225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:40.741059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:40.794184 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409f6d85-d48f-44f6-bd33-7897f8623993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:40.795232 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409f6d85-d48f-44f6-bd33-7897f8623993 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:40.795720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409f6d85-d48f-44f6-bd33-7897f8623993 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:40.796194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409f6d85-d48f-44f6-bd33-7897f8623993 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:40.855670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 88/168] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:40.856516 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14089046-6833-43c0-b652-6ebdc2325225 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:40.940258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409f6d85-d48f-44f6-bd33-7897f8623993 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:40.941754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409f6d85-d48f-44f6-bd33-7897f8623993 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:41.067613 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:41.085872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 81/169] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 653 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:41.086719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409f6d85-d48f-44f6-bd33-7897f8623993 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:41.090414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14089046-6833-43c0-b652-6ebdc2325225 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:41.144734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b81ebc-ccba-49b9-bc94-f2d1c726ff05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:41.145478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b81ebc-ccba-49b9-bc94-f2d1c726ff05 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:41.145770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b81ebc-ccba-49b9-bc94-f2d1c726ff05 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:41.146048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b81ebc-ccba-49b9-bc94-f2d1c726ff05 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:41.206351 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409f6d85-d48f-44f6-bd33-7897f8623993 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:41.211567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:41.246177 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661ce696-6c1a-4a5f-ae00-6b0e7fff8d97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:41.246841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661ce696-6c1a-4a5f-ae00-6b0e7fff8d97 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:41.247131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661ce696-6c1a-4a5f-ae00-6b0e7fff8d97 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:41.247407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661ce696-6c1a-4a5f-ae00-6b0e7fff8d97 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:41.260227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661ce696-6c1a-4a5f-ae00-6b0e7fff8d97 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-661ce696-6c1a-4a5f-ae00-6b0e7fff8d97 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:41.266105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:41.266963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:41.267730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 82/170] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:41.268372 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661ce696-6c1a-4a5f-ae00-6b0e7fff8d97 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:41.364973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:41.373462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b81ebc-ccba-49b9-bc94-f2d1c726ff05 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:41.374972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b81ebc-ccba-49b9-bc94-f2d1c726ff05 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:41.459839 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661ce696-6c1a-4a5f-ae00-6b0e7fff8d97 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:41.497021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd8fd55-c9d5-4492-953f-e0c3dcbb1f0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:41.497681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd8fd55-c9d5-4492-953f-e0c3dcbb1f0a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:41.497971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd8fd55-c9d5-4492-953f-e0c3dcbb1f0a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:41.498255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd8fd55-c9d5-4492-953f-e0c3dcbb1f0a None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:41.499464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd8fd55-c9d5-4492-953f-e0c3dcbb1f0a None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:41.500960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd8fd55-c9d5-4492-953f-e0c3dcbb1f0a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:41.504342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 83/171] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:41 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:41.505275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd8fd55-c9d5-4492-953f-e0c3dcbb1f0a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:41.616220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd8fd55-c9d5-4492-953f-e0c3dcbb1f0a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:41.631278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:41.655267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1a0a3-104e-414d-adbb-34b581f7a51b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:41.655923 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1a0a3-104e-414d-adbb-34b581f7a51b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:41.656386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1a0a3-104e-414d-adbb-34b581f7a51b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:41.656862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1a0a3-104e-414d-adbb-34b581f7a51b None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:41.658375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae1a0a3-104e-414d-adbb-34b581f7a51b None admin] RBAC: Adding query filter params (name=member) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:41.658817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae1a0a3-104e-414d-adbb-34b581f7a51b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:41.660522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae1a0a3-104e-414d-adbb-34b581f7a51b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:41.669755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 84/172] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:41.670872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1a0a3-104e-414d-adbb-34b581f7a51b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:41.699822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 89/173] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 844 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:41.710019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b81ebc-ccba-49b9-bc94-f2d1c726ff05 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:41.782234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1a0a3-104e-414d-adbb-34b581f7a51b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:41.804964 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:41.843460 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5d52fd-db9f-4eb2-a6b9-b9dd8ebfccd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:41.844105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5d52fd-db9f-4eb2-a6b9-b9dd8ebfccd2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:41.844397 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5d52fd-db9f-4eb2-a6b9-b9dd8ebfccd2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:41.844708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5d52fd-db9f-4eb2-a6b9-b9dd8ebfccd2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:41.845520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5d52fd-db9f-4eb2-a6b9-b9dd8ebfccd2 None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d, user.id=cfc9fe47ed4740a0aaddec545dbaca1a) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:41.845841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5d52fd-db9f-4eb2-a6b9-b9dd8ebfccd2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:41.847372 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5d52fd-db9f-4eb2-a6b9-b9dd8ebfccd2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:41.863505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 85/174] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:41 2022] GET /identity/v3/role_assignments?user.id=cfc9fe47ed4740a0aaddec545dbaca1a&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:41.864480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5d52fd-db9f-4eb2-a6b9-b9dd8ebfccd2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:41.888408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b81ebc-ccba-49b9-bc94-f2d1c726ff05 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:41.933098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f725871-3c56-4e50-9cb0-5ebefacff025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:41.935113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f725871-3c56-4e50-9cb0-5ebefacff025 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:41.935619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f725871-3c56-4e50-9cb0-5ebefacff025 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:41.936153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f725871-3c56-4e50-9cb0-5ebefacff025 None admin] PATH_INFO: `/v3/users/0c530b748ab741c89390231f1b4f34c9` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:41.937766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f725871-3c56-4e50-9cb0-5ebefacff025 None admin] RBAC: Authorizing `identity:get_user(user_id=0c530b748ab741c89390231f1b4f34c9)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:41.939202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:41.939901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f725871-3c56-4e50-9cb0-5ebefacff025 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:41.943121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 90/175] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:41 2022] GET /identity/v3/users/0c530b748ab741c89390231f1b4f34c9 => generated 275 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:41.944742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cfdaa5-31df-4033-b180-46ac19d4945d None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:41.945238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cfdaa5-31df-4033-b180-46ac19d4945d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:41.945718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cfdaa5-31df-4033-b180-46ac19d4945d None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:41.948014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 91/176] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:41.952775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9fb748-2caf-4d2c-a4ae-2668d2ccf6a8 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:41.952775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9fb748-2caf-4d2c-a4ae-2668d2ccf6a8 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:41.952775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9fb748-2caf-4d2c-a4ae-2668d2ccf6a8 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:41.954360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 92/177] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:41.955039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9fb748-2caf-4d2c-a4ae-2668d2ccf6a8 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:42.027113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5d52fd-db9f-4eb2-a6b9-b9dd8ebfccd2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:42.072480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ccfbcf-421c-4c84-a933-7b3a0259f0e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:42.072480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ccfbcf-421c-4c84-a933-7b3a0259f0e9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.072480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ccfbcf-421c-4c84-a933-7b3a0259f0e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.072837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ccfbcf-421c-4c84-a933-7b3a0259f0e9 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.075719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ccfbcf-421c-4c84-a933-7b3a0259f0e9 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:42.077312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ccfbcf-421c-4c84-a933-7b3a0259f0e9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:42.079149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 86/178] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:41 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:42.084640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37293510-0585-4ebf-9b57-5fd9b6e2ee0e None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.084640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37293510-0585-4ebf-9b57-5fd9b6e2ee0e None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.084640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37293510-0585-4ebf-9b57-5fd9b6e2ee0e None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.084640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 87/179] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:42.084640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411247bf-f8d7-4092-9d8d-50da238d2e3a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.084640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411247bf-f8d7-4092-9d8d-50da238d2e3a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.085059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411247bf-f8d7-4092-9d8d-50da238d2e3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.105344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-411247bf-f8d7-4092-9d8d-50da238d2e3a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:42.106572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-411247bf-f8d7-4092-9d8d-50da238d2e3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:42.109219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9fb748-2caf-4d2c-a4ae-2668d2ccf6a8 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:42.144207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:42.167408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c1d7c0-bf7e-487d-9a27-0272984c2811 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:42.168381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c1d7c0-bf7e-487d-9a27-0272984c2811 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.169357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c1d7c0-bf7e-487d-9a27-0272984c2811 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.169840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c1d7c0-bf7e-487d-9a27-0272984c2811 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.182758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 88/180] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:42.184149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7cb04b-3b25-4b8d-ae19-cbedf2a2b804 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.184442 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7cb04b-3b25-4b8d-ae19-cbedf2a2b804 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.184747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7cb04b-3b25-4b8d-ae19-cbedf2a2b804 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.203094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-bc7cb04b-3b25-4b8d-ae19-cbedf2a2b804 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:42.204291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-bc7cb04b-3b25-4b8d-ae19-cbedf2a2b804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:42.235115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:42.250163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c1d7c0-bf7e-487d-9a27-0272984c2811 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:42.251642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c1d7c0-bf7e-487d-9a27-0272984c2811 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:42.274259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 89/181] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:42.275506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675877a-96ad-4a57-9940-d463b3fad12d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.275803 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675877a-96ad-4a57-9940-d463b3fad12d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.276084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7675877a-96ad-4a57-9940-d463b3fad12d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.294241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7675877a-96ad-4a57-9940-d463b3fad12d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:42.295371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7675877a-96ad-4a57-9940-d463b3fad12d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:42.364497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 90/182] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:42.369058 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 93/183] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:42.369727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7675877a-96ad-4a57-9940-d463b3fad12d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:42.370306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d239af1-295e-4d97-a4d7-c77cb5c674b7 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.370592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d239af1-295e-4d97-a4d7-c77cb5c674b7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.370865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d239af1-295e-4d97-a4d7-c77cb5c674b7 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.375009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 94/184] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:42 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:42.375009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d239af1-295e-4d97-a4d7-c77cb5c674b7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:42.399635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:42.471347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:42.478257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d239af1-295e-4d97-a4d7-c77cb5c674b7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:42.478404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7675877a-96ad-4a57-9940-d463b3fad12d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:42.511688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45884087-82cb-4b47-b9a5-a8f2346e4f03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:42.512309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45884087-82cb-4b47-b9a5-a8f2346e4f03 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.512478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4166e917-2820-4dd8-96b8-1219ac43aa29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:42.512586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45884087-82cb-4b47-b9a5-a8f2346e4f03 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.512898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45884087-82cb-4b47-b9a5-a8f2346e4f03 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.513160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4166e917-2820-4dd8-96b8-1219ac43aa29 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.513442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4166e917-2820-4dd8-96b8-1219ac43aa29 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.513699 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45884087-82cb-4b47-b9a5-a8f2346e4f03 None admin] RBAC: Adding query filter params (role.id=28ddf4e7cb7f4139a3b6a79077418921, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d, user.id=0c530b748ab741c89390231f1b4f34c9) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:42.513787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4166e917-2820-4dd8-96b8-1219ac43aa29 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.514017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45884087-82cb-4b47-b9a5-a8f2346e4f03 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:42.514880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4166e917-2820-4dd8-96b8-1219ac43aa29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:42.515457 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45884087-82cb-4b47-b9a5-a8f2346e4f03 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:42.516292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4166e917-2820-4dd8-96b8-1219ac43aa29 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:42.517943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 95/185] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:42.519442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4166e917-2820-4dd8-96b8-1219ac43aa29 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:42.522125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 91/186] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:42 2022] GET /identity/v3/role_assignments?user.id=0c530b748ab741c89390231f1b4f34c9&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=28ddf4e7cb7f4139a3b6a79077418921 => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:42.522828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45884087-82cb-4b47-b9a5-a8f2346e4f03 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:42.629739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:42.640368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4166e917-2820-4dd8-96b8-1219ac43aa29 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:42.645378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45884087-82cb-4b47-b9a5-a8f2346e4f03 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:42.684110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f464f0-1a5f-40fd-ac24-90956498cbae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:42.684497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f464f0-1a5f-40fd-ac24-90956498cbae None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.684815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f464f0-1a5f-40fd-ac24-90956498cbae None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.685757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f464f0-1a5f-40fd-ac24-90956498cbae None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.698659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8507ea-84d0-4675-95b2-a74f37cf4d37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:42.698923 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f464f0-1a5f-40fd-ac24-90956498cbae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:42.699387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8507ea-84d0-4675-95b2-a74f37cf4d37 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.699683 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8507ea-84d0-4675-95b2-a74f37cf4d37 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.699966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8507ea-84d0-4675-95b2-a74f37cf4d37 None admin] PATH_INFO: `/v3/projects/69ad7185f26745f4bfa315f5ad3baebf` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.701218 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8507ea-84d0-4675-95b2-a74f37cf4d37 None admin] RBAC: Authorizing `identity:get_project(project_id=69ad7185f26745f4bfa315f5ad3baebf)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:42.702673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8507ea-84d0-4675-95b2-a74f37cf4d37 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-91f464f0-1a5f-40fd-ac24-90956498cbae None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:42.703888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:42.707660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:42.708765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:42.709491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 92/187] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:42 2022] GET /identity/v3/projects/69ad7185f26745f4bfa315f5ad3baebf => generated 312 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:42.709571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 96/188] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:33:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:42.714686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78436c29-eafc-4642-a0de-5dffaf15b82f None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.714686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78436c29-eafc-4642-a0de-5dffaf15b82f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.714875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f464f0-1a5f-40fd-ac24-90956498cbae None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:42.714968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78436c29-eafc-4642-a0de-5dffaf15b82f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.726135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-78436c29-eafc-4642-a0de-5dffaf15b82f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:42.727344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-78436c29-eafc-4642-a0de-5dffaf15b82f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:42.762096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:42.807245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 93/189] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:42.808070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78436c29-eafc-4642-a0de-5dffaf15b82f None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:42.822415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f464f0-1a5f-40fd-ac24-90956498cbae None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:42.857783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b309107a-b170-45eb-a18b-ce7987fd87cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:42.858432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b309107a-b170-45eb-a18b-ce7987fd87cd None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.858719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b309107a-b170-45eb-a18b-ce7987fd87cd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.858997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b309107a-b170-45eb-a18b-ce7987fd87cd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.868928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b309107a-b170-45eb-a18b-ce7987fd87cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-b309107a-b170-45eb-a18b-ce7987fd87cd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:42.873833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:42.874707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:42.875471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 97/190] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:33:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:42.876357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b309107a-b170-45eb-a18b-ce7987fd87cd None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:42.889081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:42.914401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78436c29-eafc-4642-a0de-5dffaf15b82f None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:42.947924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecee655-b2c5-4b28-b3af-e3b3a79662fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:42.948545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecee655-b2c5-4b28-b3af-e3b3a79662fa None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:42.948867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecee655-b2c5-4b28-b3af-e3b3a79662fa None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:42.949149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ecee655-b2c5-4b28-b3af-e3b3a79662fa None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:42.949837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecee655-b2c5-4b28-b3af-e3b3a79662fa None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:42.951279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ecee655-b2c5-4b28-b3af-e3b3a79662fa None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:42.973734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 94/191] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:33:42 2022] POST /identity/v3/users => generated 297 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:42.974551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecee655-b2c5-4b28-b3af-e3b3a79662fa None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:42.991925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:42.994393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b309107a-b170-45eb-a18b-ce7987fd87cd None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.037525 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15558c1f-344e-4ea2-bafa-172697c146a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:43.038177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15558c1f-344e-4ea2-bafa-172697c146a4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:43.038466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15558c1f-344e-4ea2-bafa-172697c146a4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:43.038749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15558c1f-344e-4ea2-bafa-172697c146a4 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:43.039497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15558c1f-344e-4ea2-bafa-172697c146a4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:43.039813 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15558c1f-344e-4ea2-bafa-172697c146a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:43.041305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15558c1f-344e-4ea2-bafa-172697c146a4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:43.051216 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 98/192] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:43.057312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15558c1f-344e-4ea2-bafa-172697c146a4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:43.096822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ecee655-b2c5-4b28-b3af-e3b3a79662fa None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.134832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc6d77f-5dd1-47b6-9a38-bae86dee9116 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:43.135852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc6d77f-5dd1-47b6-9a38-bae86dee9116 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:43.136134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc6d77f-5dd1-47b6-9a38-bae86dee9116 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:43.136407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc6d77f-5dd1-47b6-9a38-bae86dee9116 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:43.137248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc6d77f-5dd1-47b6-9a38-bae86dee9116 None admin] RBAC: Adding query filter params (role.id=9e77a799dd224c1480f51a54f3aed529, scope.project.id=69ad7185f26745f4bfa315f5ad3baebf, user.id=0c530b748ab741c89390231f1b4f34c9) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:43.137559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc6d77f-5dd1-47b6-9a38-bae86dee9116 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:43.139001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc6d77f-5dd1-47b6-9a38-bae86dee9116 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:43.139649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:43.146629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 95/193] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:42 2022] GET /identity/v3/role_assignments?user.id=0c530b748ab741c89390231f1b4f34c9&scope.project.id=69ad7185f26745f4bfa315f5ad3baebf&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:43.147451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc6d77f-5dd1-47b6-9a38-bae86dee9116 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:43.175291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15558c1f-344e-4ea2-bafa-172697c146a4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.222338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f2f7a-061a-416c-8ff4-8ec6885a6513 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:43.223553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f2f7a-061a-416c-8ff4-8ec6885a6513 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:43.223849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f2f7a-061a-416c-8ff4-8ec6885a6513 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:43.224177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f2f7a-061a-416c-8ff4-8ec6885a6513 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:43.238445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5f2f7a-061a-416c-8ff4-8ec6885a6513 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-7e5f2f7a-061a-416c-8ff4-8ec6885a6513 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:43.244053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:43.245242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:43.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 99/194] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:43 2022] GET /identity/v3/roles/reader => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:43.249662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f2f7a-061a-416c-8ff4-8ec6885a6513 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:43.258799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:43.290020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc6d77f-5dd1-47b6-9a38-bae86dee9116 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.344132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9575a-b545-46ab-8d8b-cdeefe3a443c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:43.345925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9575a-b545-46ab-8d8b-cdeefe3a443c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:43.347535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9575a-b545-46ab-8d8b-cdeefe3a443c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:43.347912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9575a-b545-46ab-8d8b-cdeefe3a443c None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:43.348685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd9575a-b545-46ab-8d8b-cdeefe3a443c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:43.349001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd9575a-b545-46ab-8d8b-cdeefe3a443c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:43.350410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd9575a-b545-46ab-8d8b-cdeefe3a443c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:43.360077 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 96/195] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:33:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:43.360800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9575a-b545-46ab-8d8b-cdeefe3a443c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:43.389518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f2f7a-061a-416c-8ff4-8ec6885a6513 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.424595 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:43.435262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e53398e-1233-4a75-b7d8-1c201125a469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:43.435970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e53398e-1233-4a75-b7d8-1c201125a469 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:43.436266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e53398e-1233-4a75-b7d8-1c201125a469 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:43.436553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e53398e-1233-4a75-b7d8-1c201125a469 None admin] PATH_INFO: `/v3/users/a392ab7f694d4621a2ddddf0ed451121` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:43.461667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e53398e-1233-4a75-b7d8-1c201125a469 None admin] RBAC: Authorizing `identity:get_user(user_id=a392ab7f694d4621a2ddddf0ed451121)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:43.464291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e53398e-1233-4a75-b7d8-1c201125a469 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:43.467220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 100/196] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:43 2022] GET /identity/v3/users/a392ab7f694d4621a2ddddf0ed451121 => generated 283 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:43.468459 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e53398e-1233-4a75-b7d8-1c201125a469 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:43.477997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9575a-b545-46ab-8d8b-cdeefe3a443c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.525166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d39e56-ba5e-410e-a993-14c2d435225e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:43.525924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d39e56-ba5e-410e-a993-14c2d435225e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:43.526281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d39e56-ba5e-410e-a993-14c2d435225e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:43.526633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d39e56-ba5e-410e-a993-14c2d435225e None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:43.527443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d39e56-ba5e-410e-a993-14c2d435225e None admin] RBAC: Adding query filter params (name=reader) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:43.527821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d39e56-ba5e-410e-a993-14c2d435225e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:43.529903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d39e56-ba5e-410e-a993-14c2d435225e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:43.543133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 97/197] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:43 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:43.544090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d39e56-ba5e-410e-a993-14c2d435225e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:43.547844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:43.618072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e53398e-1233-4a75-b7d8-1c201125a469 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.657242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f284edde-ef6d-420b-af2c-b2473a9a1539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:43.658246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f284edde-ef6d-420b-af2c-b2473a9a1539 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:43.658850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f284edde-ef6d-420b-af2c-b2473a9a1539 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:43.660013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f284edde-ef6d-420b-af2c-b2473a9a1539 None admin] PATH_INFO: `/v3/users/a392ab7f694d4621a2ddddf0ed451121` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:43.661410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f284edde-ef6d-420b-af2c-b2473a9a1539 None admin] RBAC: Authorizing `identity:get_user(user_id=a392ab7f694d4621a2ddddf0ed451121)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:43.663076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f284edde-ef6d-420b-af2c-b2473a9a1539 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:43.665315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 101/198] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:43 2022] GET /identity/v3/users/a392ab7f694d4621a2ddddf0ed451121 => generated 283 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:43.670936 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f284edde-ef6d-420b-af2c-b2473a9a1539 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:43.674080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d39e56-ba5e-410e-a993-14c2d435225e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.710063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f326f46-f118-4cc7-b2ce-6ab4f9e18429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:43.710617 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f326f46-f118-4cc7-b2ce-6ab4f9e18429 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:43.710901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f326f46-f118-4cc7-b2ce-6ab4f9e18429 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:43.711173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f326f46-f118-4cc7-b2ce-6ab4f9e18429 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:43.739175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:43.782802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f326f46-f118-4cc7-b2ce-6ab4f9e18429 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:43.785061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f326f46-f118-4cc7-b2ce-6ab4f9e18429 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:43.787007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:43.788337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f284edde-ef6d-420b-af2c-b2473a9a1539 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.829925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7203715-8948-4f69-9301-401353cdd35b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:43.831640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7203715-8948-4f69-9301-401353cdd35b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:43.832189 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7203715-8948-4f69-9301-401353cdd35b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:43.832716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7203715-8948-4f69-9301-401353cdd35b None admin] PATH_INFO: `/v3/users/c514bf23041040d39377c6c9fc195a04` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:43.834246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7203715-8948-4f69-9301-401353cdd35b None admin] RBAC: Authorizing `identity:get_user(user_id=c514bf23041040d39377c6c9fc195a04)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:43.835977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7203715-8948-4f69-9301-401353cdd35b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:43.838304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 102/199] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:43 2022] GET /identity/v3/users/c514bf23041040d39377c6c9fc195a04 => generated 289 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:43.839320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7203715-8948-4f69-9301-401353cdd35b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:43.890689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 98/200] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:43.891575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f326f46-f118-4cc7-b2ce-6ab4f9e18429 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:43.952578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7203715-8948-4f69-9301-401353cdd35b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.982592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:43.994327 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a0ca55-08be-45d9-82ce-500f8c9435c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:43.995305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f326f46-f118-4cc7-b2ce-6ab4f9e18429 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:43.995497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a0ca55-08be-45d9-82ce-500f8c9435c8 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:43.995983 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a0ca55-08be-45d9-82ce-500f8c9435c8 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:43.996563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a0ca55-08be-45d9-82ce-500f8c9435c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:43.997015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:44.031668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84697f3-4ecf-4015-a404-ff468bf44f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:44.032304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84697f3-4ecf-4015-a404-ff468bf44f96 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:44.032590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84697f3-4ecf-4015-a404-ff468bf44f96 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:44.032921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84697f3-4ecf-4015-a404-ff468bf44f96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:44.074602 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a0ca55-08be-45d9-82ce-500f8c9435c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:44.076503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a0ca55-08be-45d9-82ce-500f8c9435c8 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:44.102765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84697f3-4ecf-4015-a404-ff468bf44f96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:44.104237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84697f3-4ecf-4015-a404-ff468bf44f96 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:44.186271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 103/201] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:44.187478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a0ca55-08be-45d9-82ce-500f8c9435c8 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:44.206827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 99/202] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:44.207782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84697f3-4ecf-4015-a404-ff468bf44f96 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:44.210062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:44.218313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:44.309731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a0ca55-08be-45d9-82ce-500f8c9435c8 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:44.314774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84697f3-4ecf-4015-a404-ff468bf44f96 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:44.351814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129c526b-1a5e-45e9-ba1c-463306663cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:44.352761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c526b-1a5e-45e9-ba1c-463306663cb9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:44.353150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c526b-1a5e-45e9-ba1c-463306663cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:44.353533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129c526b-1a5e-45e9-ba1c-463306663cb9 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:44.354830 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129c526b-1a5e-45e9-ba1c-463306663cb9 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:44.356046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678b0c71-e18a-4011-ab14-8496d02fae91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:44.356503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129c526b-1a5e-45e9-ba1c-463306663cb9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:44.357010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b0c71-e18a-4011-ab14-8496d02fae91 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:44.357453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b0c71-e18a-4011-ab14-8496d02fae91 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:44.357867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678b0c71-e18a-4011-ab14-8496d02fae91 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:44.358904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 100/203] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:44 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:44.359340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678b0c71-e18a-4011-ab14-8496d02fae91 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:44.359874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129c526b-1a5e-45e9-ba1c-463306663cb9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:44.361423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-678b0c71-e18a-4011-ab14-8496d02fae91 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:44.364725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 104/204] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:44 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:44.365690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678b0c71-e18a-4011-ab14-8496d02fae91 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:44.442395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:44.478086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129c526b-1a5e-45e9-ba1c-463306663cb9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:44.502765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:44.513278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678b0c71-e18a-4011-ab14-8496d02fae91 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:44.515630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fad69ab-cf3d-4cc9-a8bc-f6b735f3f829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:44.516479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fad69ab-cf3d-4cc9-a8bc-f6b735f3f829 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:44.516895 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fad69ab-cf3d-4cc9-a8bc-f6b735f3f829 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:44.517285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fad69ab-cf3d-4cc9-a8bc-f6b735f3f829 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:44.519228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fad69ab-cf3d-4cc9-a8bc-f6b735f3f829 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:44.524133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fad69ab-cf3d-4cc9-a8bc-f6b735f3f829 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:44.524133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 101/205] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:44 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:44.524133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fad69ab-cf3d-4cc9-a8bc-f6b735f3f829 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:44.560865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b1c3c-e04e-4281-86c4-12e392bf09d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:44.561726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b1c3c-e04e-4281-86c4-12e392bf09d6 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:44.562158 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b1c3c-e04e-4281-86c4-12e392bf09d6 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:44.562580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b1c3c-e04e-4281-86c4-12e392bf09d6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:44.563517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b1c3c-e04e-4281-86c4-12e392bf09d6 None admin] RBAC: Adding query filter params (role.id=28ddf4e7cb7f4139a3b6a79077418921, scope.project.id=7a2a97f3a55842ac8262f51748e570c2, user.id=a392ab7f694d4621a2ddddf0ed451121) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:44.563975 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b1c3c-e04e-4281-86c4-12e392bf09d6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:44.566148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b1c3c-e04e-4281-86c4-12e392bf09d6 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:44.572562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 105/206] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:44 2022] GET /identity/v3/role_assignments?user.id=a392ab7f694d4621a2ddddf0ed451121&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=28ddf4e7cb7f4139a3b6a79077418921 => generated 259 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:44.573616 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b1c3c-e04e-4281-86c4-12e392bf09d6 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:44.662344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fad69ab-cf3d-4cc9-a8bc-f6b735f3f829 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:44.693128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b1c3c-e04e-4281-86c4-12e392bf09d6 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:44.712907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b53c5e-3b14-4ba5-ad37-21221344677b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:44.713544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b53c5e-3b14-4ba5-ad37-21221344677b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:44.713834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b53c5e-3b14-4ba5-ad37-21221344677b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:44.714112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b53c5e-3b14-4ba5-ad37-21221344677b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:44.714919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b53c5e-3b14-4ba5-ad37-21221344677b None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=7a2a97f3a55842ac8262f51748e570c2, user.id=a392ab7f694d4621a2ddddf0ed451121) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:44.715236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b53c5e-3b14-4ba5-ad37-21221344677b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:44.717013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b53c5e-3b14-4ba5-ad37-21221344677b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:44.723229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 102/207] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:44 2022] GET /identity/v3/role_assignments?user.id=a392ab7f694d4621a2ddddf0ed451121&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:44.743925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2658080-eff3-4e21-86a0-6c1dbfc27810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:44.744569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2658080-eff3-4e21-86a0-6c1dbfc27810 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:44.744938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2658080-eff3-4e21-86a0-6c1dbfc27810 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:44.745225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2658080-eff3-4e21-86a0-6c1dbfc27810 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:44.746143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2658080-eff3-4e21-86a0-6c1dbfc27810 None admin] RBAC: Adding query filter params (role.id=f382f86b952e4b869017bbf06295b028, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d, user.id=c514bf23041040d39377c6c9fc195a04) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:44.746469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2658080-eff3-4e21-86a0-6c1dbfc27810 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:44.748048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2658080-eff3-4e21-86a0-6c1dbfc27810 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:44.761565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 106/208] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:44 2022] GET /identity/v3/role_assignments?user.id=c514bf23041040d39377c6c9fc195a04&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=f382f86b952e4b869017bbf06295b028 => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:45.378949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ab8eab-9fc5-43e8-8305-00cece5190ff None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:45.379425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ab8eab-9fc5-43e8-8305-00cece5190ff None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:45.379819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ab8eab-9fc5-43e8-8305-00cece5190ff None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:45.381583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 103/209] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:45.386859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5167efc-9c5e-44bf-a238-102e600fd630 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:45.387231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5167efc-9c5e-44bf-a238-102e600fd630 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:45.387592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5167efc-9c5e-44bf-a238-102e600fd630 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:45.411118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c5167efc-9c5e-44bf-a238-102e600fd630 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:45.412640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c5167efc-9c5e-44bf-a238-102e600fd630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:45.431274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:45.454103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357d83f4-0a57-4cf7-b127-fdfd1b093f02 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:45.454394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357d83f4-0a57-4cf7-b127-fdfd1b093f02 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:45.454678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357d83f4-0a57-4cf7-b127-fdfd1b093f02 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:45.456620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 104/210] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:45.461891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545696eb-1a9d-4d60-b094-012112c1da6a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:45.462408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545696eb-1a9d-4d60-b094-012112c1da6a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:45.462884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545696eb-1a9d-4d60-b094-012112c1da6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:45.469620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:45.484324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-545696eb-1a9d-4d60-b094-012112c1da6a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:45.486494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-545696eb-1a9d-4d60-b094-012112c1da6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:45.487310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 107/211] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:45.510975 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5167efc-9c5e-44bf-a238-102e600fd630 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:45.560085 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 105/212] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:45.592937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545696eb-1a9d-4d60-b094-012112c1da6a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:45.637274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5167efc-9c5e-44bf-a238-102e600fd630 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:45.715337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31de30e3-2eea-4ab4-b415-3a1506851e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:45.716340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31de30e3-2eea-4ab4-b415-3a1506851e8a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:45.717288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31de30e3-2eea-4ab4-b415-3a1506851e8a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:45.718529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31de30e3-2eea-4ab4-b415-3a1506851e8a None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:45.721271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31de30e3-2eea-4ab4-b415-3a1506851e8a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:45.723469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31de30e3-2eea-4ab4-b415-3a1506851e8a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:45.728286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 108/213] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:45.744859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31de30e3-2eea-4ab4-b415-3a1506851e8a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:45.785532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:45.789464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545696eb-1a9d-4d60-b094-012112c1da6a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:45.849632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:45.858392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da91f7fc-0492-4f7d-a6cd-8a80d8313f27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:45.859293 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da91f7fc-0492-4f7d-a6cd-8a80d8313f27 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:45.859745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da91f7fc-0492-4f7d-a6cd-8a80d8313f27 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:45.860218 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da91f7fc-0492-4f7d-a6cd-8a80d8313f27 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:45.873694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da91f7fc-0492-4f7d-a6cd-8a80d8313f27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-da91f7fc-0492-4f7d-a6cd-8a80d8313f27 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:45.879399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:45.880281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:45.881097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:45.886486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 106/214] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:45.894022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da91f7fc-0492-4f7d-a6cd-8a80d8313f27 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:45.908674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31de30e3-2eea-4ab4-b415-3a1506851e8a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:45.993469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7681f48-6e39-423d-a00b-bcd653c591e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:45.994127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7681f48-6e39-423d-a00b-bcd653c591e4 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:45.994418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7681f48-6e39-423d-a00b-bcd653c591e4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:45.994704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7681f48-6e39-423d-a00b-bcd653c591e4 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:45.995830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7681f48-6e39-423d-a00b-bcd653c591e4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:46.003921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7681f48-6e39-423d-a00b-bcd653c591e4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:46.036628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da91f7fc-0492-4f7d-a6cd-8a80d8313f27 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:46.038023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 109/215] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:33:45 2022] POST /identity/v3/users => generated 297 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:46.109333 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8547933e-8857-4431-81a6-035fd8f6fa83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:46.110041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8547933e-8857-4431-81a6-035fd8f6fa83 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:46.110336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8547933e-8857-4431-81a6-035fd8f6fa83 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:46.110617 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8547933e-8857-4431-81a6-035fd8f6fa83 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:46.111373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8547933e-8857-4431-81a6-035fd8f6fa83 None admin] RBAC: Adding query filter params (name=member) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:46.111688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8547933e-8857-4431-81a6-035fd8f6fa83 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:46.113764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8547933e-8857-4431-81a6-035fd8f6fa83 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:46.123412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 107/216] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:46.129064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7681f48-6e39-423d-a00b-bcd653c591e4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:46.218001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:46.244232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7681f48-6e39-423d-a00b-bcd653c591e4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:46.279969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ab7f69-a699-4176-aab9-c34e4bfaca77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:46.280645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ab7f69-a699-4176-aab9-c34e4bfaca77 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:46.280939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ab7f69-a699-4176-aab9-c34e4bfaca77 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:46.281219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ab7f69-a699-4176-aab9-c34e4bfaca77 None admin] PATH_INFO: `/v3/users/3dc1c353e5064b59bd611c7c60fd7bf3` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:46.296942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ab7f69-a699-4176-aab9-c34e4bfaca77 None admin] RBAC: Authorizing `identity:get_user(user_id=3dc1c353e5064b59bd611c7c60fd7bf3)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:46.298695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ab7f69-a699-4176-aab9-c34e4bfaca77 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:46.300797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 110/217] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:46 2022] GET /identity/v3/users/3dc1c353e5064b59bd611c7c60fd7bf3 => generated 297 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:46.306124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8547933e-8857-4431-81a6-035fd8f6fa83 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:46.373534 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:46.423790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8547933e-8857-4431-81a6-035fd8f6fa83 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:46.460908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c42a64-f595-44cd-a1ea-7b569964ad3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:46.461835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c42a64-f595-44cd-a1ea-7b569964ad3c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:46.462303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c42a64-f595-44cd-a1ea-7b569964ad3c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:46.462787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c42a64-f595-44cd-a1ea-7b569964ad3c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:46.543260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c42a64-f595-44cd-a1ea-7b569964ad3c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:46.544825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c42a64-f595-44cd-a1ea-7b569964ad3c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:46.615570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:46.683833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 108/218] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 29 04:33:46.689111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ab7f69-a699-4176-aab9-c34e4bfaca77 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:46.820010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ab7f69-a699-4176-aab9-c34e4bfaca77 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:46.864324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3b9443-ee97-46cf-b57e-be5e19f8650f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:46.865008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b9443-ee97-46cf-b57e-be5e19f8650f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:46.865296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b9443-ee97-46cf-b57e-be5e19f8650f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:46.865576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3b9443-ee97-46cf-b57e-be5e19f8650f None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:46.866804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3b9443-ee97-46cf-b57e-be5e19f8650f None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:46.868259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3b9443-ee97-46cf-b57e-be5e19f8650f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:46.870274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 111/219] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:46 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:46.876174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c42a64-f595-44cd-a1ea-7b569964ad3c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:46.971708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a819f2b-758d-4684-b1db-34d686e8940d None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:46.972644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a819f2b-758d-4684-b1db-34d686e8940d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:46.972940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a819f2b-758d-4684-b1db-34d686e8940d None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:46.974558 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 112/220] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:33:46.979720 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0442e1f1-29bc-4100-a202-5a3524ae124a None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:46.980257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0442e1f1-29bc-4100-a202-5a3524ae124a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:46.980805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0442e1f1-29bc-4100-a202-5a3524ae124a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:46.989993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:47.001745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0442e1f1-29bc-4100-a202-5a3524ae124a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:47.004554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0442e1f1-29bc-4100-a202-5a3524ae124a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:47.017854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c42a64-f595-44cd-a1ea-7b569964ad3c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:47.055074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adc3c94-5114-4bff-b929-efa6af644e07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:47.055739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc3c94-5114-4bff-b929-efa6af644e07 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:47.056031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc3c94-5114-4bff-b929-efa6af644e07 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:47.056320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc3c94-5114-4bff-b929-efa6af644e07 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:47.057172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adc3c94-5114-4bff-b929-efa6af644e07 None admin] RBAC: Adding query filter params (role.id=9e77a799dd224c1480f51a54f3aed529, scope.project.id=7a2a97f3a55842ac8262f51748e570c2, user.id=3dc1c353e5064b59bd611c7c60fd7bf3) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:47.057468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adc3c94-5114-4bff-b929-efa6af644e07 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:47.058915 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adc3c94-5114-4bff-b929-efa6af644e07 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:47.068662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 109/221] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:46 2022] GET /identity/v3/role_assignments?user.id=3dc1c353e5064b59bd611c7c60fd7bf3&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 259 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:47.068662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454da75-1b2b-4d55-a22a-a332b16a3461 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:47.068662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454da75-1b2b-4d55-a22a-a332b16a3461 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:47.068662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454da75-1b2b-4d55-a22a-a332b16a3461 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:47.068662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 110/222] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:47.073875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37c45e-78dc-4e51-a6ab-04674da96079 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:47.074123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37c45e-78dc-4e51-a6ab-04674da96079 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:47.074359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce37c45e-78dc-4e51-a6ab-04674da96079 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:47.107172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-ce37c45e-78dc-4e51-a6ab-04674da96079 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:47.108412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ce37c45e-78dc-4e51-a6ab-04674da96079 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:47.122715 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 113/223] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:47.124898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:47.148522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0442e1f1-29bc-4100-a202-5a3524ae124a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:47.247301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 111/224] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:47.278227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0442e1f1-29bc-4100-a202-5a3524ae124a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:47.289293 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce37c45e-78dc-4e51-a6ab-04674da96079 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:47.319163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:47.326829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d888f9d-55fa-43cb-b206-e89249bddc7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:47.327748 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d888f9d-55fa-43cb-b206-e89249bddc7d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:47.328229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d888f9d-55fa-43cb-b206-e89249bddc7d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:47.328791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d888f9d-55fa-43cb-b206-e89249bddc7d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:47.340984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d888f9d-55fa-43cb-b206-e89249bddc7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-5d888f9d-55fa-43cb-b206-e89249bddc7d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:47.346355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:47.348458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:47.349251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:47.351883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 114/225] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:33:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:33:47.357859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d888f9d-55fa-43cb-b206-e89249bddc7d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:47.404858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce37c45e-78dc-4e51-a6ab-04674da96079 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:47.442893 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510f768c-153d-4230-bbb4-dd138201ee23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:47.443547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510f768c-153d-4230-bbb4-dd138201ee23 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:47.443899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510f768c-153d-4230-bbb4-dd138201ee23 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:47.444111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510f768c-153d-4230-bbb4-dd138201ee23 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:47.463173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510f768c-153d-4230-bbb4-dd138201ee23 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-510f768c-153d-4230-bbb4-dd138201ee23 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:47.468225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:47.469138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:47.469898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:47.477627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 112/226] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:33:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:47.485125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510f768c-153d-4230-bbb4-dd138201ee23 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:47.500188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d888f9d-55fa-43cb-b206-e89249bddc7d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:47.511669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:47.538924 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8da331-63af-432f-916b-5c75ac46b5ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:47.539905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8da331-63af-432f-916b-5c75ac46b5ce None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:47.540418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8da331-63af-432f-916b-5c75ac46b5ce None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:47.540918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8da331-63af-432f-916b-5c75ac46b5ce None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:47.541893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8da331-63af-432f-916b-5c75ac46b5ce None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:47.542410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8da331-63af-432f-916b-5c75ac46b5ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:47.543990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8da331-63af-432f-916b-5c75ac46b5ce None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:47.553670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 115/227] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:33:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:47.559652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8da331-63af-432f-916b-5c75ac46b5ce None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:47.625117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:47.670014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8da331-63af-432f-916b-5c75ac46b5ce None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:47.707051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311cbd7e-6184-41a8-8616-815b403d313b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:47.708111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311cbd7e-6184-41a8-8616-815b403d313b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:47.708697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311cbd7e-6184-41a8-8616-815b403d313b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:47.709176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311cbd7e-6184-41a8-8616-815b403d313b None admin] PATH_INFO: `/v3/users/a392ab7f694d4621a2ddddf0ed451121` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:47.710699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311cbd7e-6184-41a8-8616-815b403d313b None admin] RBAC: Authorizing `identity:get_user(user_id=a392ab7f694d4621a2ddddf0ed451121)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:47.712552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-311cbd7e-6184-41a8-8616-815b403d313b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:47.714608 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 116/228] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:47 2022] GET /identity/v3/users/a392ab7f694d4621a2ddddf0ed451121 => generated 283 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:47.720054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311cbd7e-6184-41a8-8616-815b403d313b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:47.746891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510f768c-153d-4230-bbb4-dd138201ee23 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:47.815110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57caee3d-c8d9-4213-b98e-b4f2d668a7e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:47.815809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57caee3d-c8d9-4213-b98e-b4f2d668a7e7 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:47.816093 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57caee3d-c8d9-4213-b98e-b4f2d668a7e7 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:47.816384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57caee3d-c8d9-4213-b98e-b4f2d668a7e7 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:47.821968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57caee3d-c8d9-4213-b98e-b4f2d668a7e7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:47.822321 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57caee3d-c8d9-4213-b98e-b4f2d668a7e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:47.829973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311cbd7e-6184-41a8-8616-815b403d313b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:47.830455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57caee3d-c8d9-4213-b98e-b4f2d668a7e7 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:47.844316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 113/229] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 359 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:47.851049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57caee3d-c8d9-4213-b98e-b4f2d668a7e7 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:47.857456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:47.868821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b24b45-24de-46d3-a91f-2f82a8a3bb90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:47.869905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b24b45-24de-46d3-a91f-2f82a8a3bb90 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:47.870387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b24b45-24de-46d3-a91f-2f82a8a3bb90 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:47.870859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b24b45-24de-46d3-a91f-2f82a8a3bb90 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:47.872293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b24b45-24de-46d3-a91f-2f82a8a3bb90 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:47.874127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b24b45-24de-46d3-a91f-2f82a8a3bb90 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:47.876576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 117/230] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:47 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:47.882562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b24b45-24de-46d3-a91f-2f82a8a3bb90 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:47.989487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b24b45-24de-46d3-a91f-2f82a8a3bb90 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:47.999605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:48.028624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01046ca-ba78-4506-898e-d36a34025de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:48.029547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01046ca-ba78-4506-898e-d36a34025de1 None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:48.029933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01046ca-ba78-4506-898e-d36a34025de1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:48.030441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01046ca-ba78-4506-898e-d36a34025de1 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2/users/a392ab7f694d4621a2ddddf0ed451121/roles/28ddf4e7cb7f4139a3b6a79077418921` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:48.032785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01046ca-ba78-4506-898e-d36a34025de1 None admin] RBAC: Authorizing `identity:create_grant(project_id=7a2a97f3a55842ac8262f51748e570c2, user_id=a392ab7f694d4621a2ddddf0ed451121, role_id=28ddf4e7cb7f4139a3b6a79077418921)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:48.034988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01046ca-ba78-4506-898e-d36a34025de1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:48.056752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 118/231] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:33:47 2022] PUT /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2/users/a392ab7f694d4621a2ddddf0ed451121/roles/28ddf4e7cb7f4139a3b6a79077418921 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:33:48.075371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57caee3d-c8d9-4213-b98e-b4f2d668a7e7 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:48.128590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcf3206-53be-4241-a880-d07ead1a3c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:48.129200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcf3206-53be-4241-a880-d07ead1a3c97 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:48.129488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcf3206-53be-4241-a880-d07ead1a3c97 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:48.129768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dcf3206-53be-4241-a880-d07ead1a3c97 None admin] PATH_INFO: `/v3/users/a392ab7f694d4621a2ddddf0ed451121` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:48.130972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dcf3206-53be-4241-a880-d07ead1a3c97 None admin] RBAC: Authorizing `identity:get_user(user_id=a392ab7f694d4621a2ddddf0ed451121)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:48.132510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dcf3206-53be-4241-a880-d07ead1a3c97 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:48.134467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 114/232] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:47 2022] GET /identity/v3/users/a392ab7f694d4621a2ddddf0ed451121 => generated 283 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:48.140048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01046ca-ba78-4506-898e-d36a34025de1 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:48.209645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:48.269555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01046ca-ba78-4506-898e-d36a34025de1 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:48.307056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1981b5fe-740b-40f0-868b-8fa32bb6d915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:48.307714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1981b5fe-740b-40f0-868b-8fa32bb6d915 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:48.308003 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1981b5fe-740b-40f0-868b-8fa32bb6d915 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:48.308287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1981b5fe-740b-40f0-868b-8fa32bb6d915 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:48.309571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1981b5fe-740b-40f0-868b-8fa32bb6d915 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:48.311161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1981b5fe-740b-40f0-868b-8fa32bb6d915 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:48.313188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 119/233] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:48 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:48.319381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcf3206-53be-4241-a880-d07ead1a3c97 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:48.432300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dcf3206-53be-4241-a880-d07ead1a3c97 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:48.467002 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e8ed53d-7762-4d2f-8d5b-1d6634a40ecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:48.467743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8ed53d-7762-4d2f-8d5b-1d6634a40ecd None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:48.468032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8ed53d-7762-4d2f-8d5b-1d6634a40ecd None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:48.468365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8ed53d-7762-4d2f-8d5b-1d6634a40ecd None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2/users/a392ab7f694d4621a2ddddf0ed451121/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:48.470381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8ed53d-7762-4d2f-8d5b-1d6634a40ecd None admin] RBAC: Authorizing `identity:create_grant(project_id=7a2a97f3a55842ac8262f51748e570c2, user_id=a392ab7f694d4621a2ddddf0ed451121, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:48.472191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e8ed53d-7762-4d2f-8d5b-1d6634a40ecd None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:48.483863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:48.490019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 115/234] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:33:48 2022] PUT /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2/users/a392ab7f694d4621a2ddddf0ed451121/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:33:48.622993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44233fc6-21f2-436c-9ccb-dbdeccef0b2a None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:48.623331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44233fc6-21f2-436c-9ccb-dbdeccef0b2a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:48.623569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44233fc6-21f2-436c-9ccb-dbdeccef0b2a None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:48.625031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 120/235] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:48.630687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9737402-dbc6-4d68-be3c-326dfa1641a2 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:48.630979 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9737402-dbc6-4d68-be3c-326dfa1641a2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:48.631259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9737402-dbc6-4d68-be3c-326dfa1641a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:48.650545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d9737402-dbc6-4d68-be3c-326dfa1641a2 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:48.651779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d9737402-dbc6-4d68-be3c-326dfa1641a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:48.730005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 116/236] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:48.756879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44233fc6-21f2-436c-9ccb-dbdeccef0b2a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:48.888552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44233fc6-21f2-436c-9ccb-dbdeccef0b2a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:48.911212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:48.920324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc20c0ad-b718-42a9-ace5-ef107b5af315 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:48.921251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc20c0ad-b718-42a9-ace5-ef107b5af315 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:48.921733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc20c0ad-b718-42a9-ace5-ef107b5af315 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:48.923893 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 117/237] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:33:48.929497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d562c4a-7202-4558-81c3-7d59434403ec None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:48.930054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d562c4a-7202-4558-81c3-7d59434403ec None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:48.930557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d562c4a-7202-4558-81c3-7d59434403ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:48.935935 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2705677-d5e5-478a-9acc-1e182ad4517f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:48.936586 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2705677-d5e5-478a-9acc-1e182ad4517f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:48.936894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2705677-d5e5-478a-9acc-1e182ad4517f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:48.937177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2705677-d5e5-478a-9acc-1e182ad4517f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:48.953752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-0d562c4a-7202-4558-81c3-7d59434403ec None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:48.957838 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0d562c4a-7202-4558-81c3-7d59434403ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:48.962947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2705677-d5e5-478a-9acc-1e182ad4517f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-a2705677-d5e5-478a-9acc-1e182ad4517f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:48.968713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:48.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:48.970364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:48.972939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 121/238] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:48 2022] GET /identity/v3/roles/reader => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:48.979442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2705677-d5e5-478a-9acc-1e182ad4517f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:49.095635 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 118/239] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:49.127585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d562c4a-7202-4558-81c3-7d59434403ec None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:49.151478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2705677-d5e5-478a-9acc-1e182ad4517f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:49.215837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:49.218911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f69859-60a2-42c5-bc1e-497167496a2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:49.219538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f69859-60a2-42c5-bc1e-497167496a2a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:49.220060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f69859-60a2-42c5-bc1e-497167496a2a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:49.220340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f69859-60a2-42c5-bc1e-497167496a2a None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:49.221091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f69859-60a2-42c5-bc1e-497167496a2a None admin] RBAC: Adding query filter params (name=reader) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:49.221396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f69859-60a2-42c5-bc1e-497167496a2a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:49.222918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f69859-60a2-42c5-bc1e-497167496a2a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:49.239194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 122/240] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:48 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:49.249719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f69859-60a2-42c5-bc1e-497167496a2a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:49.274401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d562c4a-7202-4558-81c3-7d59434403ec None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:49.314615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dd13f5-f905-422c-a763-75a2906c62e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:49.315582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dd13f5-f905-422c-a763-75a2906c62e3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:49.316042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dd13f5-f905-422c-a763-75a2906c62e3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:49.317663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84dd13f5-f905-422c-a763-75a2906c62e3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:49.318992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dd13f5-f905-422c-a763-75a2906c62e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:49.321325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84dd13f5-f905-422c-a763-75a2906c62e3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:49.323730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 119/241] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:49.330305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dd13f5-f905-422c-a763-75a2906c62e3 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:49.357065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:49.450047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84dd13f5-f905-422c-a763-75a2906c62e3 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:49.484962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:49.498576 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ed737b-1e2a-497f-b0ee-2979c13bfc65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:49.499212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ed737b-1e2a-497f-b0ee-2979c13bfc65 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:49.499500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ed737b-1e2a-497f-b0ee-2979c13bfc65 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:49.499780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ed737b-1e2a-497f-b0ee-2979c13bfc65 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:49.500492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ed737b-1e2a-497f-b0ee-2979c13bfc65 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:49.502657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ed737b-1e2a-497f-b0ee-2979c13bfc65 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:49.512856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f69859-60a2-42c5-bc1e-497167496a2a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:49.538687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 120/242] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:33:49 2022] POST /identity/v3/users => generated 293 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:49.572233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e71649-2138-46d2-992e-d80de359cd91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:49.572938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e71649-2138-46d2-992e-d80de359cd91 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:49.573234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e71649-2138-46d2-992e-d80de359cd91 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:49.573528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e71649-2138-46d2-992e-d80de359cd91 None admin] PATH_INFO: `/v3/users/b3f60e6182734cf1b5e2d01562c54748` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:49.597872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e71649-2138-46d2-992e-d80de359cd91 None admin] RBAC: Authorizing `identity:get_user(user_id=b3f60e6182734cf1b5e2d01562c54748)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:49.599587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e71649-2138-46d2-992e-d80de359cd91 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:49.601732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 123/243] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:49 2022] GET /identity/v3/users/b3f60e6182734cf1b5e2d01562c54748 => generated 297 bytes in 352 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:49.607402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ed737b-1e2a-497f-b0ee-2979c13bfc65 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:49.721235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ed737b-1e2a-497f-b0ee-2979c13bfc65 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:49.728138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28befae-6cce-4fde-a990-fab74fcdfac0 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:49.728786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28befae-6cce-4fde-a990-fab74fcdfac0 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:49.729273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28befae-6cce-4fde-a990-fab74fcdfac0 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:49.731080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 124/244] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:49.736171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d740f11-8596-4194-95ae-c146bc63d904 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:49.736794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d740f11-8596-4194-95ae-c146bc63d904 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:49.737312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d740f11-8596-4194-95ae-c146bc63d904 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:49.758887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9224ca4f-304b-45d2-a815-55124096d460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:49.759518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9224ca4f-304b-45d2-a815-55124096d460 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:49.759800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9224ca4f-304b-45d2-a815-55124096d460 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:49.760079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9224ca4f-304b-45d2-a815-55124096d460 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:49.769270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4d740f11-8596-4194-95ae-c146bc63d904 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:49.772879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4d740f11-8596-4194-95ae-c146bc63d904 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:49.842301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:49.866772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9224ca4f-304b-45d2-a815-55124096d460 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:49.866937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 125/245] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:49.869015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9224ca4f-304b-45d2-a815-55124096d460 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:49.892551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d740f11-8596-4194-95ae-c146bc63d904 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:49.928925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:49.998039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 121/246] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:50.004685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9224ca4f-304b-45d2-a815-55124096d460 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:50.022009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d740f11-8596-4194-95ae-c146bc63d904 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:50.064684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36155e6f-57b3-413a-8496-9a3b29ae11ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:50.065527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36155e6f-57b3-413a-8496-9a3b29ae11ae None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:50.065842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36155e6f-57b3-413a-8496-9a3b29ae11ae None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:50.066118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36155e6f-57b3-413a-8496-9a3b29ae11ae None admin] PATH_INFO: `/v3/roles/member` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:50.078412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36155e6f-57b3-413a-8496-9a3b29ae11ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-36155e6f-57b3-413a-8496-9a3b29ae11ae None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:50.084133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:50.085221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:50.085995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 126/247] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:33:50.094015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36155e6f-57b3-413a-8496-9a3b29ae11ae None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:50.113899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:50.137001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9224ca4f-304b-45d2-a815-55124096d460 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:50.179865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66018fcd-e413-4acb-ba50-2d8e0e795b72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:50.181644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66018fcd-e413-4acb-ba50-2d8e0e795b72 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:50.181946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66018fcd-e413-4acb-ba50-2d8e0e795b72 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:50.182331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66018fcd-e413-4acb-ba50-2d8e0e795b72 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:50.183552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66018fcd-e413-4acb-ba50-2d8e0e795b72 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:50.185226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66018fcd-e413-4acb-ba50-2d8e0e795b72 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:50.187310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 122/248] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:50 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:50.194281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66018fcd-e413-4acb-ba50-2d8e0e795b72 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:50.196457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:50.210680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36155e6f-57b3-413a-8496-9a3b29ae11ae None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:50.248298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35b1da8-3fb0-4bee-9098-8a5e9b4797d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:50.248965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35b1da8-3fb0-4bee-9098-8a5e9b4797d7 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:50.249256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35b1da8-3fb0-4bee-9098-8a5e9b4797d7 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:50.249538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35b1da8-3fb0-4bee-9098-8a5e9b4797d7 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:50.250264 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35b1da8-3fb0-4bee-9098-8a5e9b4797d7 None admin] RBAC: Adding query filter params (name=member) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:50.250575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35b1da8-3fb0-4bee-9098-8a5e9b4797d7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:50.253307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35b1da8-3fb0-4bee-9098-8a5e9b4797d7 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:50.262741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 127/249] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:50.270600 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35b1da8-3fb0-4bee-9098-8a5e9b4797d7 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:50.325760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66018fcd-e413-4acb-ba50-2d8e0e795b72 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:50.362737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a858cd-9c6c-4bba-a0cc-e1474fdf97d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:50.363410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a858cd-9c6c-4bba-a0cc-e1474fdf97d9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:50.363698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a858cd-9c6c-4bba-a0cc-e1474fdf97d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:50.363979 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a858cd-9c6c-4bba-a0cc-e1474fdf97d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:50.364898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a858cd-9c6c-4bba-a0cc-e1474fdf97d9 None admin] RBAC: Adding query filter params (role.id=f382f86b952e4b869017bbf06295b028, scope.project.id=7a2a97f3a55842ac8262f51748e570c2, user.id=b3f60e6182734cf1b5e2d01562c54748) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:50.365218 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a858cd-9c6c-4bba-a0cc-e1474fdf97d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:50.366674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a858cd-9c6c-4bba-a0cc-e1474fdf97d9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:50.373711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 123/250] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:50 2022] GET /identity/v3/role_assignments?user.id=b3f60e6182734cf1b5e2d01562c54748&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=f382f86b952e4b869017bbf06295b028 => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:50.394743 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35b1da8-3fb0-4bee-9098-8a5e9b4797d7 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:50.432703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2249590e-a526-4611-986d-dc07d306a907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:50.433359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2249590e-a526-4611-986d-dc07d306a907 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:50.433640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2249590e-a526-4611-986d-dc07d306a907 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:50.433915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2249590e-a526-4611-986d-dc07d306a907 None admin] PATH_INFO: `/v3/users/3dc1c353e5064b59bd611c7c60fd7bf3` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:50.435121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2249590e-a526-4611-986d-dc07d306a907 None admin] RBAC: Authorizing `identity:get_user(user_id=3dc1c353e5064b59bd611c7c60fd7bf3)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:50.436592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2249590e-a526-4611-986d-dc07d306a907 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:50.438681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 128/251] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:50 2022] GET /identity/v3/users/3dc1c353e5064b59bd611c7c60fd7bf3 => generated 297 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:50.445715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a858cd-9c6c-4bba-a0cc-e1474fdf97d9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:50.473009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:50.600703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a858cd-9c6c-4bba-a0cc-e1474fdf97d9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:50.641591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9807012-815e-4303-9e20-e5ef0169d238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:50.642251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9807012-815e-4303-9e20-e5ef0169d238 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:50.642543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9807012-815e-4303-9e20-e5ef0169d238 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:50.642824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9807012-815e-4303-9e20-e5ef0169d238 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:50.644524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9807012-815e-4303-9e20-e5ef0169d238 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:50.646066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9807012-815e-4303-9e20-e5ef0169d238 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:50.648003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 124/252] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:50 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:50.653734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2249590e-a526-4611-986d-dc07d306a907 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:50.678350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:50.789703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2249590e-a526-4611-986d-dc07d306a907 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:50.835528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5250641d-b443-4a0d-88fa-4dcafeef1a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:50.836221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5250641d-b443-4a0d-88fa-4dcafeef1a3b None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:50.836510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5250641d-b443-4a0d-88fa-4dcafeef1a3b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:50.836819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5250641d-b443-4a0d-88fa-4dcafeef1a3b None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2/users/3dc1c353e5064b59bd611c7c60fd7bf3/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:50.838824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5250641d-b443-4a0d-88fa-4dcafeef1a3b None admin] RBAC: Authorizing `identity:create_grant(project_id=7a2a97f3a55842ac8262f51748e570c2, user_id=3dc1c353e5064b59bd611c7c60fd7bf3, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:50.840309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5250641d-b443-4a0d-88fa-4dcafeef1a3b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:50.858200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 129/253] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:33:50 2022] PUT /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2/users/3dc1c353e5064b59bd611c7c60fd7bf3/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:33:50.962611 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d55e923-d68e-4d9e-b7a6-832c23a7fe5c None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:50.962940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d55e923-d68e-4d9e-b7a6-832c23a7fe5c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:50.963104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d55e923-d68e-4d9e-b7a6-832c23a7fe5c None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:50.964857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 125/254] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:50.970157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2cc71-b30f-43b6-925b-bce5fca032cc None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:50.970628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2cc71-b30f-43b6-925b-bce5fca032cc None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:50.970726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2cc71-b30f-43b6-925b-bce5fca032cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:50.994346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-8ca2cc71-b30f-43b6-925b-bce5fca032cc None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:50.995603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8ca2cc71-b30f-43b6-925b-bce5fca032cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:51.090105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 130/255] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:51.125327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d55e923-d68e-4d9e-b7a6-832c23a7fe5c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:51.190923 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e31955b-a2c5-4ac1-a511-82bc7a7cc02f None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:51.191814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e31955b-a2c5-4ac1-a511-82bc7a7cc02f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:51.192304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e31955b-a2c5-4ac1-a511-82bc7a7cc02f None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:51.193901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 131/256] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:51.199118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b26e555-52cb-4493-b2ad-2a3989cc4733 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:51.199687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b26e555-52cb-4493-b2ad-2a3989cc4733 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:51.200168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b26e555-52cb-4493-b2ad-2a3989cc4733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:51.219904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4b26e555-52cb-4493-b2ad-2a3989cc4733 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:51.222003 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4b26e555-52cb-4493-b2ad-2a3989cc4733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:51.229536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:51.231892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:51.289312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d55e923-d68e-4d9e-b7a6-832c23a7fe5c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:51.307843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 132/257] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:51.326330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b26e555-52cb-4493-b2ad-2a3989cc4733 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:51.341302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a9ee57-0bd1-420e-8f2e-e9de0d2b5b69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:51.342053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a9ee57-0bd1-420e-8f2e-e9de0d2b5b69 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:51.342295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a9ee57-0bd1-420e-8f2e-e9de0d2b5b69 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:51.342580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a9ee57-0bd1-420e-8f2e-e9de0d2b5b69 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:51.359310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a9ee57-0bd1-420e-8f2e-e9de0d2b5b69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-00a9ee57-0bd1-420e-8f2e-e9de0d2b5b69 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:51.365886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:51.366756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:33:51.367518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:51.373028 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 126/258] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:33:51 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:51.374371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a9ee57-0bd1-420e-8f2e-e9de0d2b5b69 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:51.437693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b26e555-52cb-4493-b2ad-2a3989cc4733 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:51.475372 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f433bb-e8ce-4187-afb6-b8a06c61f0a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:51.476352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f433bb-e8ce-4187-afb6-b8a06c61f0a4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:51.476890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f433bb-e8ce-4187-afb6-b8a06c61f0a4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:51.477372 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f433bb-e8ce-4187-afb6-b8a06c61f0a4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:51.484920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:51.496969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f433bb-e8ce-4187-afb6-b8a06c61f0a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-32f433bb-e8ce-4187-afb6-b8a06c61f0a4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:51.506615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:51.507502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:33:51.508262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:51.511272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 133/259] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:33:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:33:51.517361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f433bb-e8ce-4187-afb6-b8a06c61f0a4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:51.622748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a9ee57-0bd1-420e-8f2e-e9de0d2b5b69 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:51.712734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c74899-9f81-4fb4-89e0-2942966c622b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:51.712734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c74899-9f81-4fb4-89e0-2942966c622b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:51.712734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c74899-9f81-4fb4-89e0-2942966c622b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:51.712734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c74899-9f81-4fb4-89e0-2942966c622b None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:51.713211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c74899-9f81-4fb4-89e0-2942966c622b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:51.713211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c74899-9f81-4fb4-89e0-2942966c622b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:51.713628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c74899-9f81-4fb4-89e0-2942966c622b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:51.735102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f433bb-e8ce-4187-afb6-b8a06c61f0a4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:51.742844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 127/260] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:33:51 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 369 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:51.756625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c74899-9f81-4fb4-89e0-2942966c622b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:51.764362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:51.795934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5afe4e0-0cd3-41d9-b554-d8f00564efea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:51.796953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5afe4e0-0cd3-41d9-b554-d8f00564efea None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:51.797579 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5afe4e0-0cd3-41d9-b554-d8f00564efea None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:51.798085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5afe4e0-0cd3-41d9-b554-d8f00564efea None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:51.799271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5afe4e0-0cd3-41d9-b554-d8f00564efea None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:51.799904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5afe4e0-0cd3-41d9-b554-d8f00564efea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:51.801554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5afe4e0-0cd3-41d9-b554-d8f00564efea None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:51.816017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 134/261] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:51.821871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5afe4e0-0cd3-41d9-b554-d8f00564efea None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:51.892885 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c74899-9f81-4fb4-89e0-2942966c622b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:51.913058 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:51.929176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4753f7cc-56cc-47ab-b14d-14ddeb66b8f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:51.929827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753f7cc-56cc-47ab-b14d-14ddeb66b8f4 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:51.930109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753f7cc-56cc-47ab-b14d-14ddeb66b8f4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:51.930384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4753f7cc-56cc-47ab-b14d-14ddeb66b8f4 None admin] PATH_INFO: `/v3/users/a392ab7f694d4621a2ddddf0ed451121` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:51.931562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4753f7cc-56cc-47ab-b14d-14ddeb66b8f4 None admin] RBAC: Authorizing `identity:get_user(user_id=a392ab7f694d4621a2ddddf0ed451121)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:51.933052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4753f7cc-56cc-47ab-b14d-14ddeb66b8f4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:51.934661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 128/262] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:51 2022] GET /identity/v3/users/a392ab7f694d4621a2ddddf0ed451121 => generated 283 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:51.940824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4753f7cc-56cc-47ab-b14d-14ddeb66b8f4 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:51.995789 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5afe4e0-0cd3-41d9-b554-d8f00564efea None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:52.016708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:52.049657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4753f7cc-56cc-47ab-b14d-14ddeb66b8f4 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:52.065432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfe606f-4f1c-40ed-a874-d643813f2292 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:52.066425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfe606f-4f1c-40ed-a874-d643813f2292 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:52.066931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfe606f-4f1c-40ed-a874-d643813f2292 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:52.067515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfe606f-4f1c-40ed-a874-d643813f2292 None admin] PATH_INFO: `/v3/users/a392ab7f694d4621a2ddddf0ed451121` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:52.069077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfe606f-4f1c-40ed-a874-d643813f2292 None admin] RBAC: Authorizing `identity:get_user(user_id=a392ab7f694d4621a2ddddf0ed451121)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:52.070748 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfe606f-4f1c-40ed-a874-d643813f2292 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:52.072983 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 135/263] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:51 2022] GET /identity/v3/users/a392ab7f694d4621a2ddddf0ed451121 => generated 283 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:52.078095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfe606f-4f1c-40ed-a874-d643813f2292 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:52.086916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48bd6aa-a7e9-4ed5-8d16-abc393dcdff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:52.087547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48bd6aa-a7e9-4ed5-8d16-abc393dcdff0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:52.087828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48bd6aa-a7e9-4ed5-8d16-abc393dcdff0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:52.093003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48bd6aa-a7e9-4ed5-8d16-abc393dcdff0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:52.182710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48bd6aa-a7e9-4ed5-8d16-abc393dcdff0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:52.184290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48bd6aa-a7e9-4ed5-8d16-abc393dcdff0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:52.277055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:52.311578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 129/264] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 29 04:33:52.314414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48bd6aa-a7e9-4ed5-8d16-abc393dcdff0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:52.346705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfe606f-4f1c-40ed-a874-d643813f2292 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:52.416284 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48bd6aa-a7e9-4ed5-8d16-abc393dcdff0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:52.416966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:52.437376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b00fcc-b514-4906-a432-88446e047226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:52.438024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b00fcc-b514-4906-a432-88446e047226 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:52.438313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b00fcc-b514-4906-a432-88446e047226 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:52.438599 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b00fcc-b514-4906-a432-88446e047226 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:52.452953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d06172d-01c7-4b4a-a342-72012cb6f67d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:52.453603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d06172d-01c7-4b4a-a342-72012cb6f67d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:52.453885 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d06172d-01c7-4b4a-a342-72012cb6f67d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:52.454161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d06172d-01c7-4b4a-a342-72012cb6f67d None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:52.455326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d06172d-01c7-4b4a-a342-72012cb6f67d None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:52.457515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d06172d-01c7-4b4a-a342-72012cb6f67d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:52.464109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 130/265] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:52 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:52.465333 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d06172d-01c7-4b4a-a342-72012cb6f67d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:52.507384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:52.546757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b00fcc-b514-4906-a432-88446e047226 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:52.548166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b00fcc-b514-4906-a432-88446e047226 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:52.567547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d06172d-01c7-4b4a-a342-72012cb6f67d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:52.602726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb2263-91e2-48af-af47-88e6bc3288d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:52.603499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb2263-91e2-48af-af47-88e6bc3288d9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:52.603880 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb2263-91e2-48af-af47-88e6bc3288d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:52.604254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bb2263-91e2-48af-af47-88e6bc3288d9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:52.605179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bb2263-91e2-48af-af47-88e6bc3288d9 None admin] RBAC: Adding query filter params (role.id=28ddf4e7cb7f4139a3b6a79077418921, scope.project.id=7a2a97f3a55842ac8262f51748e570c2, user.id=a392ab7f694d4621a2ddddf0ed451121) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:52.605595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bb2263-91e2-48af-af47-88e6bc3288d9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:52.607143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bb2263-91e2-48af-af47-88e6bc3288d9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:52.619773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 131/266] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:52 2022] GET /identity/v3/role_assignments?user.id=a392ab7f694d4621a2ddddf0ed451121&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=28ddf4e7cb7f4139a3b6a79077418921 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:52.715720 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 136/267] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 638 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 29 04:33:52.721518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb2263-91e2-48af-af47-88e6bc3288d9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:52.871701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bb2263-91e2-48af-af47-88e6bc3288d9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:52.874406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:52.908377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7812bd-cb28-451c-aad6-29b86958d217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:52.910016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7812bd-cb28-451c-aad6-29b86958d217 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:52.910305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7812bd-cb28-451c-aad6-29b86958d217 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:52.910587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7812bd-cb28-451c-aad6-29b86958d217 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:52.911794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7812bd-cb28-451c-aad6-29b86958d217 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:52.913420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7812bd-cb28-451c-aad6-29b86958d217 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:52.915455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 132/268] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:52 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:52.921023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b00fcc-b514-4906-a432-88446e047226 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:52.971259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:53.065749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b00fcc-b514-4906-a432-88446e047226 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:53.121258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd5f986-8d0f-4162-b9bc-4329e99e6289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:53.122211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd5f986-8d0f-4162-b9bc-4329e99e6289 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.122650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd5f986-8d0f-4162-b9bc-4329e99e6289 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.123086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd5f986-8d0f-4162-b9bc-4329e99e6289 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.124035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd5f986-8d0f-4162-b9bc-4329e99e6289 None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=7a2a97f3a55842ac8262f51748e570c2, user.id=a392ab7f694d4621a2ddddf0ed451121) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:53.124497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd5f986-8d0f-4162-b9bc-4329e99e6289 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:53.126144 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd5f986-8d0f-4162-b9bc-4329e99e6289 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:53.137577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 137/269] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:52 2022] GET /identity/v3/role_assignments?user.id=a392ab7f694d4621a2ddddf0ed451121&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 608 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:53.263412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b69b7f-60b5-4e1c-b35f-d5bbd438f9af None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.263921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b69b7f-60b5-4e1c-b35f-d5bbd438f9af None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.264197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b69b7f-60b5-4e1c-b35f-d5bbd438f9af None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.266123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 133/270] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:53.276429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902df0d-4261-4ee2-8617-178afacf4692 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.277085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902df0d-4261-4ee2-8617-178afacf4692 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.277552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902df0d-4261-4ee2-8617-178afacf4692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.303753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-a902df0d-4261-4ee2-8617-178afacf4692 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:53.305310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a902df0d-4261-4ee2-8617-178afacf4692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:53.310181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eaedc7-1366-402f-9eff-228214bc750a None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.310476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eaedc7-1366-402f-9eff-228214bc750a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.310767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eaedc7-1366-402f-9eff-228214bc750a None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.312268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 134/271] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:53.318395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405ada9f-673f-48e6-9efd-e3e7cdfbde0f None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.318687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405ada9f-673f-48e6-9efd-e3e7cdfbde0f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.318972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405ada9f-673f-48e6-9efd-e3e7cdfbde0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.343027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-405ada9f-673f-48e6-9efd-e3e7cdfbde0f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:53.344360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-405ada9f-673f-48e6-9efd-e3e7cdfbde0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:53.411922 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 138/272] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:53.439356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a902df0d-4261-4ee2-8617-178afacf4692 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:53.440731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 135/273] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:53.458010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2171245-4c45-4296-9c1a-3df21e55aac3 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.458306 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2171245-4c45-4296-9c1a-3df21e55aac3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.458595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2171245-4c45-4296-9c1a-3df21e55aac3 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.460221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 136/274] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:33:53.466520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24089988-8cf8-4f10-bd1b-88b2161a5225 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.466813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24089988-8cf8-4f10-bd1b-88b2161a5225 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.467097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24089988-8cf8-4f10-bd1b-88b2161a5225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.474279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:53.486836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-24089988-8cf8-4f10-bd1b-88b2161a5225 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:53.488242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-24089988-8cf8-4f10-bd1b-88b2161a5225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:53.570295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a902df0d-4261-4ee2-8617-178afacf4692 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:53.581264 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 137/275] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:53.581264 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24089988-8cf8-4f10-bd1b-88b2161a5225 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:53.609444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3367eb0d-a3ce-481d-8880-9b924adf9da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:53.610095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3367eb0d-a3ce-481d-8880-9b924adf9da2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.610379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3367eb0d-a3ce-481d-8880-9b924adf9da2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.610659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3367eb0d-a3ce-481d-8880-9b924adf9da2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.617777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:53.621870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3367eb0d-a3ce-481d-8880-9b924adf9da2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-3367eb0d-a3ce-481d-8880-9b924adf9da2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:53.626772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:53.628265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:53.629257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:53.633486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 139/276] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:33:53.634289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3367eb0d-a3ce-481d-8880-9b924adf9da2 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:53.705114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24089988-8cf8-4f10-bd1b-88b2161a5225 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:53.721190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:53.741901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00bbeda-7608-4e3e-baaa-a4fd5c492ba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:53.742557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00bbeda-7608-4e3e-baaa-a4fd5c492ba2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.742847 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00bbeda-7608-4e3e-baaa-a4fd5c492ba2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.743043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3367eb0d-a3ce-481d-8880-9b924adf9da2 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:53.743161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00bbeda-7608-4e3e-baaa-a4fd5c492ba2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.744274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00bbeda-7608-4e3e-baaa-a4fd5c492ba2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:53.746063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00bbeda-7608-4e3e-baaa-a4fd5c492ba2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:53.752047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 138/277] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:53.752047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00bbeda-7608-4e3e-baaa-a4fd5c492ba2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:53.784100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b94ddb4-1ed4-4a5f-bac6-b99ba9dcd9dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:53.784778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b94ddb4-1ed4-4a5f-bac6-b99ba9dcd9dd None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.785066 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b94ddb4-1ed4-4a5f-bac6-b99ba9dcd9dd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.785351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b94ddb4-1ed4-4a5f-bac6-b99ba9dcd9dd None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.796005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b94ddb4-1ed4-4a5f-bac6-b99ba9dcd9dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-0b94ddb4-1ed4-4a5f-bac6-b99ba9dcd9dd None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:53.801904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:53.802769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 140/278] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:53 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:53.804152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b94ddb4-1ed4-4a5f-bac6-b99ba9dcd9dd None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:53.865099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:53.869957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00bbeda-7608-4e3e-baaa-a4fd5c492ba2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:53.908470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6d66ca-f0e2-45eb-8e91-38aeb78518c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:53.909423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6d66ca-f0e2-45eb-8e91-38aeb78518c9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.909711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6d66ca-f0e2-45eb-8e91-38aeb78518c9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.909991 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6d66ca-f0e2-45eb-8e91-38aeb78518c9 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.910716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6d66ca-f0e2-45eb-8e91-38aeb78518c9 None admin] RBAC: Adding query filter params (name=member) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:53.911025 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6d66ca-f0e2-45eb-8e91-38aeb78518c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:53.912399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6d66ca-f0e2-45eb-8e91-38aeb78518c9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:53.913055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b94ddb4-1ed4-4a5f-bac6-b99ba9dcd9dd None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:53.920924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 139/279] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:53.926782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6d66ca-f0e2-45eb-8e91-38aeb78518c9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:53.951421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35734c57-f835-47da-a889-7fbd3b1692d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:53.952046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35734c57-f835-47da-a889-7fbd3b1692d9 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:53.952331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35734c57-f835-47da-a889-7fbd3b1692d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:53.952625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35734c57-f835-47da-a889-7fbd3b1692d9 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:53.953312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35734c57-f835-47da-a889-7fbd3b1692d9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:53.954746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35734c57-f835-47da-a889-7fbd3b1692d9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:53.976194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:53.992757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 141/280] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:33:53 2022] POST /identity/v3/users => generated 293 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:33:53.992757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35734c57-f835-47da-a889-7fbd3b1692d9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:54.039238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6d66ca-f0e2-45eb-8e91-38aeb78518c9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:54.080092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3fea86-2fae-4253-97c0-7da7eb966b40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:54.080831 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3fea86-2fae-4253-97c0-7da7eb966b40 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:54.081199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3fea86-2fae-4253-97c0-7da7eb966b40 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:54.081512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3fea86-2fae-4253-97c0-7da7eb966b40 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:54.082381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3fea86-2fae-4253-97c0-7da7eb966b40 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:54.082698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3fea86-2fae-4253-97c0-7da7eb966b40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:54.084138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3fea86-2fae-4253-97c0-7da7eb966b40 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:54.093265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 140/281] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:53 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:54.094399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a1ff50-03c5-47e8-86a5-ac620d877a1f None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:54.094693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a1ff50-03c5-47e8-86a5-ac620d877a1f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:54.094977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a1ff50-03c5-47e8-86a5-ac620d877a1f None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:54.096859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 141/282] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:54.102044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a1ff50-03c5-47e8-86a5-ac620d877a1f None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:54.122768 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35734c57-f835-47da-a889-7fbd3b1692d9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:54.131775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:54.166670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06a034b-b06a-48f8-a2b9-2024f7dbf3c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:54.167332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06a034b-b06a-48f8-a2b9-2024f7dbf3c9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:54.168821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06a034b-b06a-48f8-a2b9-2024f7dbf3c9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:54.168821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06a034b-b06a-48f8-a2b9-2024f7dbf3c9 None admin] PATH_INFO: `/v3/users/2ffb9fd008ba47fe9092cbc91419ff02` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:54.186416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06a034b-b06a-48f8-a2b9-2024f7dbf3c9 None admin] RBAC: Authorizing `identity:get_user(user_id=2ffb9fd008ba47fe9092cbc91419ff02)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:54.188098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06a034b-b06a-48f8-a2b9-2024f7dbf3c9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:54.189842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 142/283] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:53 2022] GET /identity/v3/users/2ffb9fd008ba47fe9092cbc91419ff02 => generated 293 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:54.192053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cda9b5b-d809-492a-b090-ca747b5729a3 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:54.192414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cda9b5b-d809-492a-b090-ca747b5729a3 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:54.192648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cda9b5b-d809-492a-b090-ca747b5729a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:54.211691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6cda9b5b-d809-492a-b090-ca747b5729a3 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:54.212910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6cda9b5b-d809-492a-b090-ca747b5729a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:54.227996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a1ff50-03c5-47e8-86a5-ac620d877a1f None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:54.277188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7ec84f-a3c1-4b48-abb5-0621d72b1e2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:54.278673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7ec84f-a3c1-4b48-abb5-0621d72b1e2d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:54.279172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7ec84f-a3c1-4b48-abb5-0621d72b1e2d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:54.279748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7ec84f-a3c1-4b48-abb5-0621d72b1e2d None admin] PATH_INFO: `/v3/users/b3f60e6182734cf1b5e2d01562c54748` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:54.280786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7ec84f-a3c1-4b48-abb5-0621d72b1e2d None admin] RBAC: Authorizing `identity:get_user(user_id=b3f60e6182734cf1b5e2d01562c54748)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:54.280968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:54.282240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7ec84f-a3c1-4b48-abb5-0621d72b1e2d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:54.284868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 142/284] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:54 2022] GET /identity/v3/users/b3f60e6182734cf1b5e2d01562c54748 => generated 297 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:54.284868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7ec84f-a3c1-4b48-abb5-0621d72b1e2d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:54.302701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 143/285] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:54.303552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cda9b5b-d809-492a-b090-ca747b5729a3 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:54.389432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:54.402646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7ec84f-a3c1-4b48-abb5-0621d72b1e2d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:54.415565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cda9b5b-d809-492a-b090-ca747b5729a3 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:54.447724 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481bc8e8-d158-4957-af86-68721d47c819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:54.448542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481bc8e8-d158-4957-af86-68721d47c819 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:54.448863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481bc8e8-d158-4957-af86-68721d47c819 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:54.449146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481bc8e8-d158-4957-af86-68721d47c819 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:54.450099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481bc8e8-d158-4957-af86-68721d47c819 None admin] RBAC: Adding query filter params (role.id=9e77a799dd224c1480f51a54f3aed529, user.id=2ffb9fd008ba47fe9092cbc91419ff02, scope.system=all) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:54.450434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481bc8e8-d158-4957-af86-68721d47c819 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:54.452221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481bc8e8-d158-4957-af86-68721d47c819 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:54.464631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 143/286] 10.222.0.40 () {56 vars in 1308 bytes} [Fri Jul 29 04:33:54 2022] GET /identity/v3/role_assignments?user.id=2ffb9fd008ba47fe9092cbc91419ff02&scope.system=all&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 226 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:54.464631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481bc8e8-d158-4957-af86-68721d47c819 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:54.480995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0154532-0860-43a6-861b-5781c637c611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:54.482021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0154532-0860-43a6-861b-5781c637c611 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:54.482521 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0154532-0860-43a6-861b-5781c637c611 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:54.482992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0154532-0860-43a6-861b-5781c637c611 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:54.484481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0154532-0860-43a6-861b-5781c637c611 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:54.486183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0154532-0860-43a6-861b-5781c637c611 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:54.490393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 144/287] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:54 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:54.495985 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0154532-0860-43a6-861b-5781c637c611 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:54.586793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481bc8e8-d158-4957-af86-68721d47c819 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:54.637666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b455e1a-e0e8-45fb-862e-f94adf32e396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:54.638383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b455e1a-e0e8-45fb-862e-f94adf32e396 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:54.638674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b455e1a-e0e8-45fb-862e-f94adf32e396 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:54.638953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b455e1a-e0e8-45fb-862e-f94adf32e396 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:54.661066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b455e1a-e0e8-45fb-862e-f94adf32e396 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-7b455e1a-e0e8-45fb-862e-f94adf32e396 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:54.667393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:54.668302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:54.669127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 144/288] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:54.676687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:54.677458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b455e1a-e0e8-45fb-862e-f94adf32e396 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:54.719738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:54.735225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0154532-0860-43a6-861b-5781c637c611 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:54.779265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df15aff8-ec94-4f61-99a1-b6b9391e3768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:54.780296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df15aff8-ec94-4f61-99a1-b6b9391e3768 None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:54.780816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df15aff8-ec94-4f61-99a1-b6b9391e3768 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:54.781351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df15aff8-ec94-4f61-99a1-b6b9391e3768 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2/users/b3f60e6182734cf1b5e2d01562c54748/roles/f382f86b952e4b869017bbf06295b028` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:54.784476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df15aff8-ec94-4f61-99a1-b6b9391e3768 None admin] RBAC: Authorizing `identity:create_grant(project_id=7a2a97f3a55842ac8262f51748e570c2, user_id=b3f60e6182734cf1b5e2d01562c54748, role_id=f382f86b952e4b869017bbf06295b028)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:54.787005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df15aff8-ec94-4f61-99a1-b6b9391e3768 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:54.805740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 145/289] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:33:54 2022] PUT /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2/users/b3f60e6182734cf1b5e2d01562c54748/roles/f382f86b952e4b869017bbf06295b028 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:33:54.867322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b455e1a-e0e8-45fb-862e-f94adf32e396 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:54.926037 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbabec1-38f0-47bc-83cb-79b14df2dfb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:54.926532 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbabec1-38f0-47bc-83cb-79b14df2dfb3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:54.926820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbabec1-38f0-47bc-83cb-79b14df2dfb3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:54.927100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbabec1-38f0-47bc-83cb-79b14df2dfb3 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:54.927855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbabec1-38f0-47bc-83cb-79b14df2dfb3 None admin] RBAC: Adding query filter params (name=member) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:54.928166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbabec1-38f0-47bc-83cb-79b14df2dfb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:54.929637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbabec1-38f0-47bc-83cb-79b14df2dfb3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:54.939561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 145/290] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:54.946153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df15aff8-ec94-4f61-99a1-b6b9391e3768 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:55.069366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df15aff8-ec94-4f61-99a1-b6b9391e3768 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:55.086344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:55.122255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585a73e4-19c9-475f-b099-09fcd602d28d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:55.122982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585a73e4-19c9-475f-b099-09fcd602d28d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:55.123501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585a73e4-19c9-475f-b099-09fcd602d28d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:55.123573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585a73e4-19c9-475f-b099-09fcd602d28d None admin] PATH_INFO: `/v3/users/3dc1c353e5064b59bd611c7c60fd7bf3` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:55.124788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585a73e4-19c9-475f-b099-09fcd602d28d None admin] RBAC: Authorizing `identity:get_user(user_id=3dc1c353e5064b59bd611c7c60fd7bf3)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:55.126245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585a73e4-19c9-475f-b099-09fcd602d28d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:55.128843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 146/291] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:54 2022] GET /identity/v3/users/3dc1c353e5064b59bd611c7c60fd7bf3 => generated 297 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:55.134255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbabec1-38f0-47bc-83cb-79b14df2dfb3 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:55.262503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:55.295295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbabec1-38f0-47bc-83cb-79b14df2dfb3 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:55.342714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcbc64a-6f17-4a4b-bde3-c139ebe6242e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:55.343264 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcbc64a-6f17-4a4b-bde3-c139ebe6242e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:55.343551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcbc64a-6f17-4a4b-bde3-c139ebe6242e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:55.343830 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcbc64a-6f17-4a4b-bde3-c139ebe6242e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:55.425498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcbc64a-6f17-4a4b-bde3-c139ebe6242e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:55.427328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcbc64a-6f17-4a4b-bde3-c139ebe6242e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:55.514182 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:55.550231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 146/292] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:55.557679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585a73e4-19c9-475f-b099-09fcd602d28d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:55.691618 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585a73e4-19c9-475f-b099-09fcd602d28d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:55.742242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ac7c04-8f72-4219-9952-33d67b5f8dcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:55.743136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac7c04-8f72-4219-9952-33d67b5f8dcb None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:55.743567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac7c04-8f72-4219-9952-33d67b5f8dcb None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:55.744045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac7c04-8f72-4219-9952-33d67b5f8dcb None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:55.745532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ac7c04-8f72-4219-9952-33d67b5f8dcb None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:55.747147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ac7c04-8f72-4219-9952-33d67b5f8dcb None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:55.748991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 147/293] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:55 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:55.755038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcbc64a-6f17-4a4b-bde3-c139ebe6242e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:55.884212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcbc64a-6f17-4a4b-bde3-c139ebe6242e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:55.926898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b161be9a-21c6-4e8b-8dbc-a60bf5e71a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:55.927453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161be9a-21c6-4e8b-8dbc-a60bf5e71a04 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:55.927741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161be9a-21c6-4e8b-8dbc-a60bf5e71a04 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:55.928021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161be9a-21c6-4e8b-8dbc-a60bf5e71a04 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:55.928847 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b161be9a-21c6-4e8b-8dbc-a60bf5e71a04 None admin] RBAC: Adding query filter params (role.id=9e77a799dd224c1480f51a54f3aed529, scope.project.id=7a2a97f3a55842ac8262f51748e570c2, user.id=3dc1c353e5064b59bd611c7c60fd7bf3) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:55.929163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b161be9a-21c6-4e8b-8dbc-a60bf5e71a04 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:55.930592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b161be9a-21c6-4e8b-8dbc-a60bf5e71a04 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:55.936879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 147/294] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:55 2022] GET /identity/v3/role_assignments?user.id=3dc1c353e5064b59bd611c7c60fd7bf3&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:56.611385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de422d87-5954-4c3f-a2e9-0745e792776b None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:56.612286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de422d87-5954-4c3f-a2e9-0745e792776b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:56.612766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de422d87-5954-4c3f-a2e9-0745e792776b None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:56.614553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 148/295] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:56.619637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6fd2a4-2100-48eb-847f-7cbcc744eeea None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:56.620083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6fd2a4-2100-48eb-847f-7cbcc744eeea None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:56.620339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6fd2a4-2100-48eb-847f-7cbcc744eeea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:56.646192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-3a6fd2a4-2100-48eb-847f-7cbcc744eeea None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:56.647418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3a6fd2a4-2100-48eb-847f-7cbcc744eeea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:56.687530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:56.703779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be14f2f2-d9e4-4300-9e5e-b4b174aa1969 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:56.704185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be14f2f2-d9e4-4300-9e5e-b4b174aa1969 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:56.704544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be14f2f2-d9e4-4300-9e5e-b4b174aa1969 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:56.706785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 149/296] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:56.711784 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d49f3db-7314-4cc8-9da5-fd29157b1722 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:56.712147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d49f3db-7314-4cc8-9da5-fd29157b1722 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:56.712516 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d49f3db-7314-4cc8-9da5-fd29157b1722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:56.726744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 148/297] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:56.733989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-2d49f3db-7314-4cc8-9da5-fd29157b1722 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:56.735285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2d49f3db-7314-4cc8-9da5-fd29157b1722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:56.749570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6fd2a4-2100-48eb-847f-7cbcc744eeea None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:56.801004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:56.813825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 150/298] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:56.837179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d49f3db-7314-4cc8-9da5-fd29157b1722 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:56.863272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6fd2a4-2100-48eb-847f-7cbcc744eeea None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:56.899147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58ab5bd-b99b-46b4-98b1-f0791d0848d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:56.900041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58ab5bd-b99b-46b4-98b1-f0791d0848d9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:56.900470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58ab5bd-b99b-46b4-98b1-f0791d0848d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:56.900946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58ab5bd-b99b-46b4-98b1-f0791d0848d9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:56.902243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58ab5bd-b99b-46b4-98b1-f0791d0848d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:56.904024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58ab5bd-b99b-46b4-98b1-f0791d0848d9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:56.905815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 149/299] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:56.911490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58ab5bd-b99b-46b4-98b1-f0791d0848d9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:56.955534 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d49f3db-7314-4cc8-9da5-fd29157b1722 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:56.964062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:56.994487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da32bc69-22dd-4e3d-938f-fe23d87d04d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:56.995540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da32bc69-22dd-4e3d-938f-fe23d87d04d9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:56.996023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da32bc69-22dd-4e3d-938f-fe23d87d04d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:56.996667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da32bc69-22dd-4e3d-938f-fe23d87d04d9 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.009537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da32bc69-22dd-4e3d-938f-fe23d87d04d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-da32bc69-22dd-4e3d-938f-fe23d87d04d9 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:57.015255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:57.016138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:57.016930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:57.020052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 151/300] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:56 2022] GET /identity/v3/roles/reader => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:33:57.021605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f91b83b-326f-4eaf-a76f-2d0fc7b2c1f4 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.022164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f91b83b-326f-4eaf-a76f-2d0fc7b2c1f4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.022646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f91b83b-326f-4eaf-a76f-2d0fc7b2c1f4 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.024339 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 152/301] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:57.026361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f91b83b-326f-4eaf-a76f-2d0fc7b2c1f4 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:57.035835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58ab5bd-b99b-46b4-98b1-f0791d0848d9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:57.071689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e87bf6-8711-4829-9337-80e08dbca35b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:57.071892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:57.072326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e87bf6-8711-4829-9337-80e08dbca35b None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.072626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e87bf6-8711-4829-9337-80e08dbca35b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.072967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e87bf6-8711-4829-9337-80e08dbca35b None admin] PATH_INFO: `/v3/groups` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.073752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e87bf6-8711-4829-9337-80e08dbca35b None admin] RBAC: Authorizing `identity:create_group()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:57.075241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e87bf6-8711-4829-9337-80e08dbca35b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:57.091402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 150/302] 10.222.0.40 () {60 vars in 1134 bytes} [Fri Jul 29 04:33:56 2022] POST /identity/v3/groups => generated 229 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:33:57.092948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe840b13-d67f-4b12-82a8-7958e959c13c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.093258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe840b13-d67f-4b12-82a8-7958e959c13c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.093561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe840b13-d67f-4b12-82a8-7958e959c13c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.112366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-fe840b13-d67f-4b12-82a8-7958e959c13c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:57.113692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fe840b13-d67f-4b12-82a8-7958e959c13c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:57.137665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f91b83b-326f-4eaf-a76f-2d0fc7b2c1f4 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:57.179342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bffcd09-cf1b-4c95-a1f1-ff20365b77c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:57.179976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bffcd09-cf1b-4c95-a1f1-ff20365b77c2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.180262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bffcd09-cf1b-4c95-a1f1-ff20365b77c2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.180540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bffcd09-cf1b-4c95-a1f1-ff20365b77c2 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.181302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bffcd09-cf1b-4c95-a1f1-ff20365b77c2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:57.181609 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bffcd09-cf1b-4c95-a1f1-ff20365b77c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:57.183023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bffcd09-cf1b-4c95-a1f1-ff20365b77c2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:57.192548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 153/303] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:57 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:33:57.193997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6bdee-bf6a-4bce-a7ab-7e117011ca89 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.194288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6bdee-bf6a-4bce-a7ab-7e117011ca89 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.194583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6bdee-bf6a-4bce-a7ab-7e117011ca89 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.195898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 154/304] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:57.196702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 151/305] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:57.200843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb6bdee-bf6a-4bce-a7ab-7e117011ca89 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:57.201758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703059f-ca5c-47d9-b17f-ebb85f7e9f98 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.202220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703059f-ca5c-47d9-b17f-ebb85f7e9f98 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.202693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7703059f-ca5c-47d9-b17f-ebb85f7e9f98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.227753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7703059f-ca5c-47d9-b17f-ebb85f7e9f98 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:57.229126 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7703059f-ca5c-47d9-b17f-ebb85f7e9f98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:57.261083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:57.320646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 152/306] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:57.320646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7703059f-ca5c-47d9-b17f-ebb85f7e9f98 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:57.320933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb6bdee-bf6a-4bce-a7ab-7e117011ca89 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:57.323133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:57.359563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84b400-5534-453a-b6f8-b1b4566b8d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:57.360214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84b400-5534-453a-b6f8-b1b4566b8d87 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.360508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84b400-5534-453a-b6f8-b1b4566b8d87 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.360820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db84b400-5534-453a-b6f8-b1b4566b8d87 None admin] PATH_INFO: `/v3/users/8f220d426c4a4f1b89dcd29589a5be4c` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.375418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84b400-5534-453a-b6f8-b1b4566b8d87 None admin] RBAC: Authorizing `identity:get_user(user_id=8f220d426c4a4f1b89dcd29589a5be4c)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:57.376858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db84b400-5534-453a-b6f8-b1b4566b8d87 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:57.378616 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 155/307] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:57 2022] GET /identity/v3/users/8f220d426c4a4f1b89dcd29589a5be4c => generated 293 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:57.379391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84b400-5534-453a-b6f8-b1b4566b8d87 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:57.436835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7703059f-ca5c-47d9-b17f-ebb85f7e9f98 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:57.476750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7426dc76-ff9e-4e42-a538-003d87227c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:57.477413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7426dc76-ff9e-4e42-a538-003d87227c33 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.477701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7426dc76-ff9e-4e42-a538-003d87227c33 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.477983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7426dc76-ff9e-4e42-a538-003d87227c33 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.489081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db84b400-5534-453a-b6f8-b1b4566b8d87 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:57.490091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7426dc76-ff9e-4e42-a538-003d87227c33 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-7426dc76-ff9e-4e42-a538-003d87227c33 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:57.495374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:57.496219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 153/308] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:57.501517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7426dc76-ff9e-4e42-a538-003d87227c33 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:57.523331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:57.532615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35456552-ddbc-4163-b377-651b0815dd8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:57.533293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35456552-ddbc-4163-b377-651b0815dd8f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.533718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35456552-ddbc-4163-b377-651b0815dd8f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.533914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35456552-ddbc-4163-b377-651b0815dd8f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.545329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35456552-ddbc-4163-b377-651b0815dd8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-35456552-ddbc-4163-b377-651b0815dd8f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:33:57.551233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:33:57.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 156/309] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:57 2022] GET /identity/v3/roles/reader => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:33:57.553342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35456552-ddbc-4163-b377-651b0815dd8f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:57.578038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:57.616810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7426dc76-ff9e-4e42-a538-003d87227c33 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:57.658993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad9b74c-835c-429f-975d-fbd4ebb1fd0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:57.659689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad9b74c-835c-429f-975d-fbd4ebb1fd0e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.659977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad9b74c-835c-429f-975d-fbd4ebb1fd0e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.660253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad9b74c-835c-429f-975d-fbd4ebb1fd0e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.661626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad9b74c-835c-429f-975d-fbd4ebb1fd0e None admin] RBAC: Adding query filter params (role.id=f382f86b952e4b869017bbf06295b028, user.id=8f220d426c4a4f1b89dcd29589a5be4c, scope.system=all) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:57.662461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad9b74c-835c-429f-975d-fbd4ebb1fd0e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:57.664439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad9b74c-835c-429f-975d-fbd4ebb1fd0e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:57.664830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35456552-ddbc-4163-b377-651b0815dd8f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:57.680631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 154/310] 10.222.0.40 () {56 vars in 1308 bytes} [Fri Jul 29 04:33:57 2022] GET /identity/v3/role_assignments?user.id=8f220d426c4a4f1b89dcd29589a5be4c&scope.system=all&role.id=f382f86b952e4b869017bbf06295b028 => generated 226 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:57.680631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad9b74c-835c-429f-975d-fbd4ebb1fd0e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:57.719663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408a15b3-0f99-44c0-8271-b1fc0a5daf19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:57.720340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408a15b3-0f99-44c0-8271-b1fc0a5daf19 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.720644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408a15b3-0f99-44c0-8271-b1fc0a5daf19 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.720929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408a15b3-0f99-44c0-8271-b1fc0a5daf19 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.722408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408a15b3-0f99-44c0-8271-b1fc0a5daf19 None admin] RBAC: Adding query filter params (name=member) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:57.722833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408a15b3-0f99-44c0-8271-b1fc0a5daf19 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:57.724400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408a15b3-0f99-44c0-8271-b1fc0a5daf19 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:57.736629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 157/311] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:57.743311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408a15b3-0f99-44c0-8271-b1fc0a5daf19 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:57.778695 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:57.798738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad9b74c-835c-429f-975d-fbd4ebb1fd0e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:57.839484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:57.840626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe0dd63-f4c5-4b09-b237-0d4b392343b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:57.841599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe0dd63-f4c5-4b09-b237-0d4b392343b2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.841902 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe0dd63-f4c5-4b09-b237-0d4b392343b2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.842691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe0dd63-f4c5-4b09-b237-0d4b392343b2 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.843560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe0dd63-f4c5-4b09-b237-0d4b392343b2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:57.843851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe0dd63-f4c5-4b09-b237-0d4b392343b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:57.845475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe0dd63-f4c5-4b09-b237-0d4b392343b2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:57.854698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 155/312] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:33:57 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:57.860568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe0dd63-f4c5-4b09-b237-0d4b392343b2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:57.861188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408a15b3-0f99-44c0-8271-b1fc0a5daf19 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:57.906411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dbca18-3d15-4ed0-b625-ff7d4beb442f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:57.906978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dbca18-3d15-4ed0-b625-ff7d4beb442f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:57.907267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dbca18-3d15-4ed0-b625-ff7d4beb442f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:57.907549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dbca18-3d15-4ed0-b625-ff7d4beb442f None admin] PATH_INFO: `/v3/users/2ffb9fd008ba47fe9092cbc91419ff02` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:57.908944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dbca18-3d15-4ed0-b625-ff7d4beb442f None admin] RBAC: Authorizing `identity:get_user(user_id=2ffb9fd008ba47fe9092cbc91419ff02)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:57.910539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dbca18-3d15-4ed0-b625-ff7d4beb442f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:57.916922 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 158/313] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:57 2022] GET /identity/v3/users/2ffb9fd008ba47fe9092cbc91419ff02 => generated 293 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:57.917801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dbca18-3d15-4ed0-b625-ff7d4beb442f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:57.993280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe0dd63-f4c5-4b09-b237-0d4b392343b2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:58.031143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5f0229-aa58-47c3-b914-15cbc17baff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:58.031821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5f0229-aa58-47c3-b914-15cbc17baff0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:58.032109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5f0229-aa58-47c3-b914-15cbc17baff0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:58.032389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee5f0229-aa58-47c3-b914-15cbc17baff0 None admin] PATH_INFO: `/v3/users/b3f60e6182734cf1b5e2d01562c54748` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:58.033700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5f0229-aa58-47c3-b914-15cbc17baff0 None admin] RBAC: Authorizing `identity:get_user(user_id=b3f60e6182734cf1b5e2d01562c54748)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:58.035593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee5f0229-aa58-47c3-b914-15cbc17baff0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:58.037626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 156/314] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:33:57 2022] GET /identity/v3/users/b3f60e6182734cf1b5e2d01562c54748 => generated 297 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:58.042815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5f0229-aa58-47c3-b914-15cbc17baff0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:58.045863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dbca18-3d15-4ed0-b625-ff7d4beb442f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:58.077383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:58.093968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e45047-29a1-4caa-abac-78edafd7fae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:58.095077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e45047-29a1-4caa-abac-78edafd7fae3 None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:58.095575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e45047-29a1-4caa-abac-78edafd7fae3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:58.096046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e45047-29a1-4caa-abac-78edafd7fae3 None admin] PATH_INFO: `/v3/system/users/2ffb9fd008ba47fe9092cbc91419ff02/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:58.097848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e45047-29a1-4caa-abac-78edafd7fae3 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=2ffb9fd008ba47fe9092cbc91419ff02, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:58.099544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e45047-29a1-4caa-abac-78edafd7fae3 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:58.110965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 159/315] 10.222.0.40 () {58 vars in 1334 bytes} [Fri Jul 29 04:33:57 2022] PUT /identity/v3/system/users/2ffb9fd008ba47fe9092cbc91419ff02/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:33:58.172679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee5f0229-aa58-47c3-b914-15cbc17baff0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:58.213297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a71233-d0eb-47d3-93a8-b29f8fd97390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:58.214367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a71233-d0eb-47d3-93a8-b29f8fd97390 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:58.214885 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a71233-d0eb-47d3-93a8-b29f8fd97390 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:58.215361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a71233-d0eb-47d3-93a8-b29f8fd97390 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:58.295210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a71233-d0eb-47d3-93a8-b29f8fd97390 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:58.297074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a71233-d0eb-47d3-93a8-b29f8fd97390 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:58.314712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:58.404376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 157/316] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:33:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:33:58.410035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e45047-29a1-4caa-abac-78edafd7fae3 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:58.427934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:58.518562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e45047-29a1-4caa-abac-78edafd7fae3 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:58.553290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e401d99-1992-47d7-b79e-8f187111daa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:58.553940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e401d99-1992-47d7-b79e-8f187111daa9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:58.554225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e401d99-1992-47d7-b79e-8f187111daa9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:58.554504 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e401d99-1992-47d7-b79e-8f187111daa9 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:58.555670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e401d99-1992-47d7-b79e-8f187111daa9 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:58.557148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e401d99-1992-47d7-b79e-8f187111daa9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:58.559096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 160/317] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:33:58 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:58.565424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a71233-d0eb-47d3-93a8-b29f8fd97390 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:58.679952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a71233-d0eb-47d3-93a8-b29f8fd97390 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:58.697542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:58.720346 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63014a74-2745-4e07-a3f2-43ae2b430347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:58.721784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63014a74-2745-4e07-a3f2-43ae2b430347 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:58.722220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63014a74-2745-4e07-a3f2-43ae2b430347 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:58.722825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63014a74-2745-4e07-a3f2-43ae2b430347 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:58.723779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63014a74-2745-4e07-a3f2-43ae2b430347 None admin] RBAC: Adding query filter params (role.id=f382f86b952e4b869017bbf06295b028, scope.project.id=7a2a97f3a55842ac8262f51748e570c2, user.id=b3f60e6182734cf1b5e2d01562c54748) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:33:58.724243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63014a74-2745-4e07-a3f2-43ae2b430347 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:58.725880 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63014a74-2745-4e07-a3f2-43ae2b430347 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:58.732768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 158/318] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:33:58 2022] GET /identity/v3/role_assignments?user.id=b3f60e6182734cf1b5e2d01562c54748&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=f382f86b952e4b869017bbf06295b028 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:59.167658 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e5b745-e247-4fef-8364-2b3137ba9d07 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:59.168178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e5b745-e247-4fef-8364-2b3137ba9d07 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:59.168253 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e5b745-e247-4fef-8364-2b3137ba9d07 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:59.169581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 161/319] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:59.175651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45ed533-6870-4ae7-abab-669d28c3166a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:59.175945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45ed533-6870-4ae7-abab-669d28c3166a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:59.176225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45ed533-6870-4ae7-abab-669d28c3166a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:59.195417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-e45ed533-6870-4ae7-abab-669d28c3166a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:59.196644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e45ed533-6870-4ae7-abab-669d28c3166a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:59.266194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 159/320] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:33:59.288940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e5b745-e247-4fef-8364-2b3137ba9d07 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:59.376897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:59.397996 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e5b745-e247-4fef-8364-2b3137ba9d07 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:59.431781 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2f6191-10d1-4029-a177-298b70583165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:59.432353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f6191-10d1-4029-a177-298b70583165 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:59.432666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f6191-10d1-4029-a177-298b70583165 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:59.432948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2f6191-10d1-4029-a177-298b70583165 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:59.434090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f6191-10d1-4029-a177-298b70583165 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:59.435562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2f6191-10d1-4029-a177-298b70583165 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:59.437392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 162/321] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:33:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:33:59.443480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45ed533-6870-4ae7-abab-669d28c3166a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:59.542594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45ed533-6870-4ae7-abab-669d28c3166a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:33:59.551998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:33:59.577641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7768ab-a88c-452c-9ec7-ef8c890238d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:33:59.578270 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7768ab-a88c-452c-9ec7-ef8c890238d4 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:59.578560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7768ab-a88c-452c-9ec7-ef8c890238d4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:59.578840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7768ab-a88c-452c-9ec7-ef8c890238d4 None admin] PATH_INFO: `/v3/groups` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:59.579513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7768ab-a88c-452c-9ec7-ef8c890238d4 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:33:59.581074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7768ab-a88c-452c-9ec7-ef8c890238d4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:33:59.597041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 160/322] 10.222.0.40 () {60 vars in 1134 bytes} [Fri Jul 29 04:33:59 2022] POST /identity/v3/groups => generated 226 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:33:59.729228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aa054e-aedf-4875-8f9b-a01a91d5a6cb None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:59.729524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aa054e-aedf-4875-8f9b-a01a91d5a6cb None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:59.729694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aa054e-aedf-4875-8f9b-a01a91d5a6cb None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:59.731229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 163/323] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:33:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:33:59.736624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbbd989-e7cf-4831-95e1-de7f145cdc8f None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:33:59.737714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbbd989-e7cf-4831-95e1-de7f145cdc8f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:33:59.737823 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbbd989-e7cf-4831-95e1-de7f145cdc8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:33:59.756703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2bbbd989-e7cf-4831-95e1-de7f145cdc8f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:33:59.757924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2bbbd989-e7cf-4831-95e1-de7f145cdc8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:33:59.832494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 161/324] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:33:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:33:59.855701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aa054e-aedf-4875-8f9b-a01a91d5a6cb None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:33:59.978163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aa054e-aedf-4875-8f9b-a01a91d5a6cb None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:00.014610 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af0a516-0d92-4e68-b355-96810abd99f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:00.015267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af0a516-0d92-4e68-b355-96810abd99f1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:00.015609 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af0a516-0d92-4e68-b355-96810abd99f1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:00.015886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af0a516-0d92-4e68-b355-96810abd99f1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:00.027736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af0a516-0d92-4e68-b355-96810abd99f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-2af0a516-0d92-4e68-b355-96810abd99f1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:00.033414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:00.035272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:00.036497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 164/325] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:33:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:00.041985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbbd989-e7cf-4831-95e1-de7f145cdc8f None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:00.151649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:00.155733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbbd989-e7cf-4831-95e1-de7f145cdc8f None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:00.195504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567bae9e-bad1-4b30-959e-1a7a1f0c1dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:00.196134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567bae9e-bad1-4b30-959e-1a7a1f0c1dde None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:00.196420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567bae9e-bad1-4b30-959e-1a7a1f0c1dde None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:00.196725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567bae9e-bad1-4b30-959e-1a7a1f0c1dde None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:00.197464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567bae9e-bad1-4b30-959e-1a7a1f0c1dde None admin] RBAC: Adding query filter params (name=reader) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:00.197772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567bae9e-bad1-4b30-959e-1a7a1f0c1dde None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:00.199163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567bae9e-bad1-4b30-959e-1a7a1f0c1dde None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:00.209164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 162/326] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:34:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:00.215154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af0a516-0d92-4e68-b355-96810abd99f1 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:00.230842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:00.279833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f7aa8b-13a6-44c2-844d-2b886dbb964d None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:00.280119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f7aa8b-13a6-44c2-844d-2b886dbb964d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:00.280329 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f7aa8b-13a6-44c2-844d-2b886dbb964d None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:00.281981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 163/327] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:34:00.287888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd02bff-408b-4a4c-a533-8227b0b4b420 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:00.288191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd02bff-408b-4a4c-a533-8227b0b4b420 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:00.288477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd02bff-408b-4a4c-a533-8227b0b4b420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:00.309154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-abd02bff-408b-4a4c-a533-8227b0b4b420 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:00.310366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-abd02bff-408b-4a4c-a533-8227b0b4b420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:00.336908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af0a516-0d92-4e68-b355-96810abd99f1 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:00.391475 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c33e8-aed8-4ce8-8aa9-2dcebd7b1145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:00.392388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c33e8-aed8-4ce8-8aa9-2dcebd7b1145 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:00.392812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c33e8-aed8-4ce8-8aa9-2dcebd7b1145 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:00.393151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c33e8-aed8-4ce8-8aa9-2dcebd7b1145 None admin] PATH_INFO: `/v3/users/8f220d426c4a4f1b89dcd29589a5be4c` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:00.397375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c33e8-aed8-4ce8-8aa9-2dcebd7b1145 None admin] RBAC: Authorizing `identity:get_user(user_id=8f220d426c4a4f1b89dcd29589a5be4c)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:00.398451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 164/328] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:00.398961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c33e8-aed8-4ce8-8aa9-2dcebd7b1145 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:00.400733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 165/329] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:34:00 2022] GET /identity/v3/users/8f220d426c4a4f1b89dcd29589a5be4c => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:00.406268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd02bff-408b-4a4c-a533-8227b0b4b420 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:00.427851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c33e8-aed8-4ce8-8aa9-2dcebd7b1145 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:00.525979 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:00.532203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:00.545128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd02bff-408b-4a4c-a533-8227b0b4b420 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:00.553500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c33e8-aed8-4ce8-8aa9-2dcebd7b1145 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:00.584987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0fc265-0e44-46fd-b450-c6ff3abbd0a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:00.586152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0fc265-0e44-46fd-b450-c6ff3abbd0a9 None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:00.586643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0fc265-0e44-46fd-b450-c6ff3abbd0a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:00.587127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0fc265-0e44-46fd-b450-c6ff3abbd0a9 None admin] PATH_INFO: `/v3/system/users/8f220d426c4a4f1b89dcd29589a5be4c/roles/f382f86b952e4b869017bbf06295b028` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:00.589600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0fc265-0e44-46fd-b450-c6ff3abbd0a9 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=8f220d426c4a4f1b89dcd29589a5be4c, role_id=f382f86b952e4b869017bbf06295b028)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:00.590790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59e5e46-cea0-40f6-a071-a5a9a02c2afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:00.591548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59e5e46-cea0-40f6-a071-a5a9a02c2afb None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:00.591786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0fc265-0e44-46fd-b450-c6ff3abbd0a9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:00.591932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59e5e46-cea0-40f6-a071-a5a9a02c2afb None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:00.592312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59e5e46-cea0-40f6-a071-a5a9a02c2afb None admin] PATH_INFO: `/v3/roles/member` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:00.603137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 165/330] 10.222.0.40 () {58 vars in 1334 bytes} [Fri Jul 29 04:34:00 2022] PUT /identity/v3/system/users/8f220d426c4a4f1b89dcd29589a5be4c/roles/f382f86b952e4b869017bbf06295b028 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:00.609987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c59e5e46-cea0-40f6-a071-a5a9a02c2afb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c59e5e46-cea0-40f6-a071-a5a9a02c2afb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:00.616665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:00.617594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:00.618363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 166/331] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:00.625765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0fc265-0e44-46fd-b450-c6ff3abbd0a9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:00.757080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0fc265-0e44-46fd-b450-c6ff3abbd0a9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:00.800081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8360153d-3ee1-411e-89ac-2baef83968a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:00.800675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8360153d-3ee1-411e-89ac-2baef83968a1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:00.801026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8360153d-3ee1-411e-89ac-2baef83968a1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:00.801379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8360153d-3ee1-411e-89ac-2baef83968a1 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:00.802172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8360153d-3ee1-411e-89ac-2baef83968a1 None admin] RBAC: Adding query filter params (name=member) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:00.802494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8360153d-3ee1-411e-89ac-2baef83968a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:00.803884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8360153d-3ee1-411e-89ac-2baef83968a1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:00.816360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 166/332] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:34:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:00.822269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59e5e46-cea0-40f6-a071-a5a9a02c2afb None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:00.936061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59e5e46-cea0-40f6-a071-a5a9a02c2afb None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:00.973635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de67785c-ca30-4c75-bbac-952f7fa1da03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:00.974291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de67785c-ca30-4c75-bbac-952f7fa1da03 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:00.974582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de67785c-ca30-4c75-bbac-952f7fa1da03 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:00.974866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de67785c-ca30-4c75-bbac-952f7fa1da03 None admin] PATH_INFO: `/v3/users/2ffb9fd008ba47fe9092cbc91419ff02` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:00.976068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de67785c-ca30-4c75-bbac-952f7fa1da03 None admin] RBAC: Authorizing `identity:get_user(user_id=2ffb9fd008ba47fe9092cbc91419ff02)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:00.977552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de67785c-ca30-4c75-bbac-952f7fa1da03 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:00.979535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 167/333] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:34:00 2022] GET /identity/v3/users/2ffb9fd008ba47fe9092cbc91419ff02 => generated 293 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:00.985244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8360153d-3ee1-411e-89ac-2baef83968a1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:01.006036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:01.104575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8360153d-3ee1-411e-89ac-2baef83968a1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:01.142779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c04143-9653-45cb-bc0a-8ac0922fd855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:01.144014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c04143-9653-45cb-bc0a-8ac0922fd855 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:01.144503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c04143-9653-45cb-bc0a-8ac0922fd855 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:01.144822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c04143-9653-45cb-bc0a-8ac0922fd855 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:01.145640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c04143-9653-45cb-bc0a-8ac0922fd855 None admin] RBAC: Adding query filter params (role.id=9e77a799dd224c1480f51a54f3aed529, user.id=2ffb9fd008ba47fe9092cbc91419ff02, scope.system=all) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:01.145962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c04143-9653-45cb-bc0a-8ac0922fd855 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:01.147549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c04143-9653-45cb-bc0a-8ac0922fd855 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:01.164510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 167/334] 10.222.0.40 () {56 vars in 1308 bytes} [Fri Jul 29 04:34:00 2022] GET /identity/v3/role_assignments?user.id=2ffb9fd008ba47fe9092cbc91419ff02&scope.system=all&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 510 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:02.255086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22844a4b-f263-4cce-9494-37b9e8e2e067 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:02.255596 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22844a4b-f263-4cce-9494-37b9e8e2e067 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:02.255670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22844a4b-f263-4cce-9494-37b9e8e2e067 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:02.257286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 168/335] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:02.272484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eccf41c-cbb3-4d04-bbd8-32c60a3aab54 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:02.273000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eccf41c-cbb3-4d04-bbd8-32c60a3aab54 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:02.273074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eccf41c-cbb3-4d04-bbd8-32c60a3aab54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:02.294984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-8eccf41c-cbb3-4d04-bbd8-32c60a3aab54 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:02.296527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8eccf41c-cbb3-4d04-bbd8-32c60a3aab54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:02.386263 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 168/336] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:34:02.413042 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22844a4b-f263-4cce-9494-37b9e8e2e067 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:02.423728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:02.539582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22844a4b-f263-4cce-9494-37b9e8e2e067 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:02.574971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cafe68-f7b8-4540-9d86-d684d64b495a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:02.575629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cafe68-f7b8-4540-9d86-d684d64b495a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:02.575969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cafe68-f7b8-4540-9d86-d684d64b495a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:02.576259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cafe68-f7b8-4540-9d86-d684d64b495a None admin] PATH_INFO: `/v3/roles/member` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:02.588087 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cafe68-f7b8-4540-9d86-d684d64b495a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c2cafe68-f7b8-4540-9d86-d684d64b495a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:02.593606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:02.594465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:02.595262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 169/337] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:02.605719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eccf41c-cbb3-4d04-bbd8-32c60a3aab54 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:02.616966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:02.739855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eccf41c-cbb3-4d04-bbd8-32c60a3aab54 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:02.781055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17b960d-6d78-4a72-972b-c622ea4ff958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:02.781743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17b960d-6d78-4a72-972b-c622ea4ff958 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:02.782036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17b960d-6d78-4a72-972b-c622ea4ff958 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:02.782320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17b960d-6d78-4a72-972b-c622ea4ff958 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:02.783061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17b960d-6d78-4a72-972b-c622ea4ff958 None admin] RBAC: Adding query filter params (name=member) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:02.783375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17b960d-6d78-4a72-972b-c622ea4ff958 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:02.785559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17b960d-6d78-4a72-972b-c622ea4ff958 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:02.794765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 169/338] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:34:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:02.805728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cafe68-f7b8-4540-9d86-d684d64b495a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:02.890707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:02.931538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cafe68-f7b8-4540-9d86-d684d64b495a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:02.941529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9030d90-7d3c-4df5-b211-0d0ae064da4d None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:02.942135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9030d90-7d3c-4df5-b211-0d0ae064da4d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:02.942622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9030d90-7d3c-4df5-b211-0d0ae064da4d None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:02.944447 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 170/339] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:02.954003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec3346-1ee3-4e9e-8ceb-bb928d4ea9dd None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:02.954597 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec3346-1ee3-4e9e-8ceb-bb928d4ea9dd None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:02.955087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ec3346-1ee3-4e9e-8ceb-bb928d4ea9dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:02.977522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9654a2c-6113-4df6-8c1f-c56a66b553bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:02.978162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9654a2c-6113-4df6-8c1f-c56a66b553bd None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:02.978448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9654a2c-6113-4df6-8c1f-c56a66b553bd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:02.978728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9654a2c-6113-4df6-8c1f-c56a66b553bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:02.996060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a1ec3346-1ee3-4e9e-8ceb-bb928d4ea9dd None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:02.997323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a1ec3346-1ee3-4e9e-8ceb-bb928d4ea9dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:03.078871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:03.094465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 171/340] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:03.113179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9654a2c-6113-4df6-8c1f-c56a66b553bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:03.114685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9654a2c-6113-4df6-8c1f-c56a66b553bd None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:03.169625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec3346-1ee3-4e9e-8ceb-bb928d4ea9dd None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:03.211404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:03.283679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 170/341] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:03.285600 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9654a2c-6113-4df6-8c1f-c56a66b553bd None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:03.306291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ec3346-1ee3-4e9e-8ceb-bb928d4ea9dd None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:03.346870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bfa211-b7f4-4f14-9938-29217cba31e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:03.347886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bfa211-b7f4-4f14-9938-29217cba31e1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:03.348377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bfa211-b7f4-4f14-9938-29217cba31e1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:03.348892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bfa211-b7f4-4f14-9938-29217cba31e1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:03.360201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bfa211-b7f4-4f14-9938-29217cba31e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-43bfa211-b7f4-4f14-9938-29217cba31e1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:03.366510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:03.367391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:03.368160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:03.370857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 172/342] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:34:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:03.377927 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bfa211-b7f4-4f14-9938-29217cba31e1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:03.455255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:03.496352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9654a2c-6113-4df6-8c1f-c56a66b553bd None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:03.512194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bfa211-b7f4-4f14-9938-29217cba31e1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:03.558543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617dc3be-03fc-4ed4-b2af-1862610683f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:03.559806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617dc3be-03fc-4ed4-b2af-1862610683f0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:03.560330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617dc3be-03fc-4ed4-b2af-1862610683f0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:03.560841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617dc3be-03fc-4ed4-b2af-1862610683f0 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:03.562308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617dc3be-03fc-4ed4-b2af-1862610683f0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:03.563086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617dc3be-03fc-4ed4-b2af-1862610683f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:03.564733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617dc3be-03fc-4ed4-b2af-1862610683f0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:03.572815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edeb51b-d196-4c75-8aa4-8d9e93dca0ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:03.573483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edeb51b-d196-4c75-8aa4-8d9e93dca0ab None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:03.573771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edeb51b-d196-4c75-8aa4-8d9e93dca0ab None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:03.574052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edeb51b-d196-4c75-8aa4-8d9e93dca0ab None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:03.575600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 173/343] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:03.576919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edeb51b-d196-4c75-8aa4-8d9e93dca0ab None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:03.578366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edeb51b-d196-4c75-8aa4-8d9e93dca0ab None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:03.580294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 171/344] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:03 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:03.581604 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617dc3be-03fc-4ed4-b2af-1862610683f0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:03.587025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edeb51b-d196-4c75-8aa4-8d9e93dca0ab None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:03.677170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:03.725894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617dc3be-03fc-4ed4-b2af-1862610683f0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:03.746529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:03.768740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c247deb-e666-483b-9563-d8c848127205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:03.769808 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c247deb-e666-483b-9563-d8c848127205 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:03.770342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c247deb-e666-483b-9563-d8c848127205 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:03.770823 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c247deb-e666-483b-9563-d8c848127205 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:03.829149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edeb51b-d196-4c75-8aa4-8d9e93dca0ab None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:03.855207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c247deb-e666-483b-9563-d8c848127205 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:03.857072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c247deb-e666-483b-9563-d8c848127205 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:03.957427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e972f37-df26-498b-a11e-8a7409d8874e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:03.958024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e972f37-df26-498b-a11e-8a7409d8874e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:03.958311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e972f37-df26-498b-a11e-8a7409d8874e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:03.958592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e972f37-df26-498b-a11e-8a7409d8874e None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:03.980673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 174/345] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:03.982631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e235ebd2-ab26-4d8b-a6e4-04512e5f5ff0 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:03.983125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e235ebd2-ab26-4d8b-a6e4-04512e5f5ff0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:03.983601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e235ebd2-ab26-4d8b-a6e4-04512e5f5ff0 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:03.985285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e972f37-df26-498b-a11e-8a7409d8874e None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:03.985931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 175/346] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:03.986980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e972f37-df26-498b-a11e-8a7409d8874e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:03.987327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e235ebd2-ab26-4d8b-a6e4-04512e5f5ff0 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:03.989110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 172/347] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:03 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 403 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:03.993566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e448b6f-1c69-4f70-aa61-8de4588e6c04 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:03.993566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e448b6f-1c69-4f70-aa61-8de4588e6c04 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:03.993786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e448b6f-1c69-4f70-aa61-8de4588e6c04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.010649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:04.036150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-9e448b6f-1c69-4f70-aa61-8de4588e6c04 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:04.037424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9e448b6f-1c69-4f70-aa61-8de4588e6c04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:04.109568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e235ebd2-ab26-4d8b-a6e4-04512e5f5ff0 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:04.120129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 173/348] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:04.121054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e448b6f-1c69-4f70-aa61-8de4588e6c04 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:04.148912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73548da3-54fb-415a-9fae-90d6e99a53d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:04.150074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73548da3-54fb-415a-9fae-90d6e99a53d2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:04.150074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73548da3-54fb-415a-9fae-90d6e99a53d2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:04.150240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73548da3-54fb-415a-9fae-90d6e99a53d2 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.151489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73548da3-54fb-415a-9fae-90d6e99a53d2 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:04.153023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73548da3-54fb-415a-9fae-90d6e99a53d2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:04.155025 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 176/349] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:03 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:04.156331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7e0be-d441-4c03-b487-0653c477316d None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:04.156634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7e0be-d441-4c03-b487-0653c477316d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:04.156924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7e0be-d441-4c03-b487-0653c477316d None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.160127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 177/350] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:04.161383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045fa8c4-136a-4337-8269-c491bc929628 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:04.161672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045fa8c4-136a-4337-8269-c491bc929628 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:04.161953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045fa8c4-136a-4337-8269-c491bc929628 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.163533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 178/351] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:04.164257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045fa8c4-136a-4337-8269-c491bc929628 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:04.169330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:04.275475 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e448b6f-1c69-4f70-aa61-8de4588e6c04 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:04.280454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:04.304393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045fa8c4-136a-4337-8269-c491bc929628 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:04.327702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b674b1-49b0-4d4f-9b60-b374786179b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:04.328226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b674b1-49b0-4d4f-9b60-b374786179b8 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:04.328454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b674b1-49b0-4d4f-9b60-b374786179b8 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:04.328901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b674b1-49b0-4d4f-9b60-b374786179b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.330292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b674b1-49b0-4d4f-9b60-b374786179b8 None admin] RBAC: Adding query filter params (group.id=9f588151b280456981cfc088281fb10b, role.id=9e77a799dd224c1480f51a54f3aed529, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:04.331365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b674b1-49b0-4d4f-9b60-b374786179b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:04.333521 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b674b1-49b0-4d4f-9b60-b374786179b8 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:04.347157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 174/352] 10.222.0.40 () {56 vars in 1376 bytes} [Fri Jul 29 04:34:04 2022] GET /identity/v3/role_assignments?group.id=9f588151b280456981cfc088281fb10b&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 260 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:04.349113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b674b1-49b0-4d4f-9b60-b374786179b8 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:04.376731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c9b965-c352-4391-8d75-9b92dcca75a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:04.378126 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c9b965-c352-4391-8d75-9b92dcca75a2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:04.378661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c9b965-c352-4391-8d75-9b92dcca75a2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:04.379124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c9b965-c352-4391-8d75-9b92dcca75a2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.392350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c9b965-c352-4391-8d75-9b92dcca75a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-e6c9b965-c352-4391-8d75-9b92dcca75a2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:04.400039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:04.401160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:04.401956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:04.404827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 179/353] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:04 2022] GET /identity/v3/roles/member => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:04.406596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf958760-34d5-4fc7-aedd-3b65ea4f938b None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:04.407092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf958760-34d5-4fc7-aedd-3b65ea4f938b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:04.407549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf958760-34d5-4fc7-aedd-3b65ea4f938b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.437662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-cf958760-34d5-4fc7-aedd-3b65ea4f938b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:04.440386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-cf958760-34d5-4fc7-aedd-3b65ea4f938b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:04.528101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:04.543516 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b674b1-49b0-4d4f-9b60-b374786179b8 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:04.548707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 180/354] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:04.551185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d9acd-8817-4eeb-8f36-85fc29e14244 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:04.551539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d9acd-8817-4eeb-8f36-85fc29e14244 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:04.551897 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d9acd-8817-4eeb-8f36-85fc29e14244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.576745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-038d9acd-8817-4eeb-8f36-85fc29e14244 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:04.579559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-038d9acd-8817-4eeb-8f36-85fc29e14244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:04.590203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c721108-0b79-40ec-afbd-014a84934adb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:04.590751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c721108-0b79-40ec-afbd-014a84934adb None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:04.591762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c721108-0b79-40ec-afbd-014a84934adb None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:04.591762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c721108-0b79-40ec-afbd-014a84934adb None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.592529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c721108-0b79-40ec-afbd-014a84934adb None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:04.594031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c721108-0b79-40ec-afbd-014a84934adb None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:04.595984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 175/355] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:04 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:04.596810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c721108-0b79-40ec-afbd-014a84934adb None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:04.694425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 181/356] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:04.696531 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038d9acd-8817-4eeb-8f36-85fc29e14244 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:04.714858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:04.798799 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c721108-0b79-40ec-afbd-014a84934adb None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:04.838190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038d9acd-8817-4eeb-8f36-85fc29e14244 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:04.845015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e237cec4-7e3d-48ee-98ae-0ab3d2392281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:04.846036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e237cec4-7e3d-48ee-98ae-0ab3d2392281 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:04.848792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e237cec4-7e3d-48ee-98ae-0ab3d2392281 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:04.849301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e237cec4-7e3d-48ee-98ae-0ab3d2392281 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.850082 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e237cec4-7e3d-48ee-98ae-0ab3d2392281 None admin] RBAC: Adding query filter params (name=member) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:04.850396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e237cec4-7e3d-48ee-98ae-0ab3d2392281 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:04.851919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e237cec4-7e3d-48ee-98ae-0ab3d2392281 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:04.864083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 176/357] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:34:04 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:04.867742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e237cec4-7e3d-48ee-98ae-0ab3d2392281 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:04.892114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87963714-bd98-46fd-8a75-4b5e4a451d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:04.893861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87963714-bd98-46fd-8a75-4b5e4a451d53 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:04.894353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87963714-bd98-46fd-8a75-4b5e4a451d53 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:04.894641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87963714-bd98-46fd-8a75-4b5e4a451d53 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:04.903860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:04.925898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87963714-bd98-46fd-8a75-4b5e4a451d53 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-87963714-bd98-46fd-8a75-4b5e4a451d53 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:04.930201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:04.931051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 29 04:34:04.931868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:04.933009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 182/358] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:04 2022] GET /identity/v3/roles/reader => generated 84 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:04.934020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87963714-bd98-46fd-8a75-4b5e4a451d53 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:05.033379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e237cec4-7e3d-48ee-98ae-0ab3d2392281 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:05.040758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:05.080941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d60d7c8-9144-41a2-b12b-5fdc2dd0ed18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:05.081502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d60d7c8-9144-41a2-b12b-5fdc2dd0ed18 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:05.081790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d60d7c8-9144-41a2-b12b-5fdc2dd0ed18 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:05.082078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d60d7c8-9144-41a2-b12b-5fdc2dd0ed18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:05.082888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d60d7c8-9144-41a2-b12b-5fdc2dd0ed18 None admin] RBAC: Adding query filter params (group.id=9f588151b280456981cfc088281fb10b, role.id=28ddf4e7cb7f4139a3b6a79077418921, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:05.083207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d60d7c8-9144-41a2-b12b-5fdc2dd0ed18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:05.084927 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d60d7c8-9144-41a2-b12b-5fdc2dd0ed18 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:05.089590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87963714-bd98-46fd-8a75-4b5e4a451d53 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:05.091692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 177/359] 10.222.0.40 () {56 vars in 1376 bytes} [Fri Jul 29 04:34:04 2022] GET /identity/v3/role_assignments?group.id=9f588151b280456981cfc088281fb10b&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=28ddf4e7cb7f4139a3b6a79077418921 => generated 260 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:05.094696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9063d6-9810-4c28-9af8-ac5ed12bc99b None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:05.094696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9063d6-9810-4c28-9af8-ac5ed12bc99b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:05.095114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9063d6-9810-4c28-9af8-ac5ed12bc99b None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:05.097275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 178/360] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:05 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:05.098029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9063d6-9810-4c28-9af8-ac5ed12bc99b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:05.155219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcc519a-7a2e-4bbd-b3ef-3e4f07da313e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:05.155986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcc519a-7a2e-4bbd-b3ef-3e4f07da313e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:05.156299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcc519a-7a2e-4bbd-b3ef-3e4f07da313e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:05.156581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcc519a-7a2e-4bbd-b3ef-3e4f07da313e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:05.169408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcc519a-7a2e-4bbd-b3ef-3e4f07da313e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-6dcc519a-7a2e-4bbd-b3ef-3e4f07da313e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:05.175322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:05.176493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:05.177487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:05.181576 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 183/361] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:34:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:05.182522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcc519a-7a2e-4bbd-b3ef-3e4f07da313e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:05.225041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:05.244052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9063d6-9810-4c28-9af8-ac5ed12bc99b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:05.284589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7f9c3b-c626-4172-972e-077bdc381d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:05.285545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7f9c3b-c626-4172-972e-077bdc381d85 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:05.285969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7f9c3b-c626-4172-972e-077bdc381d85 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:05.286414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7f9c3b-c626-4172-972e-077bdc381d85 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:05.311107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcc519a-7a2e-4bbd-b3ef-3e4f07da313e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:05.348696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:05.361704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61979b6-9f77-4c9d-aa67-ccb545a6bfe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:05.362668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61979b6-9f77-4c9d-aa67-ccb545a6bfe5 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:05.362668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61979b6-9f77-4c9d-aa67-ccb545a6bfe5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:05.362950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61979b6-9f77-4c9d-aa67-ccb545a6bfe5 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:05.363712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61979b6-9f77-4c9d-aa67-ccb545a6bfe5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:05.364075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61979b6-9f77-4c9d-aa67-ccb545a6bfe5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:05.365552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61979b6-9f77-4c9d-aa67-ccb545a6bfe5 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:05.375723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 184/362] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:34:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:05.378058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1187efa-84dc-45eb-a88e-cac0bb987439 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:05.378570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1187efa-84dc-45eb-a88e-cac0bb987439 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:05.378816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1187efa-84dc-45eb-a88e-cac0bb987439 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:05.400321 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7f9c3b-c626-4172-972e-077bdc381d85 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:05.401858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-e1187efa-84dc-45eb-a88e-cac0bb987439 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:05.402002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7f9c3b-c626-4172-972e-077bdc381d85 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:05.403326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e1187efa-84dc-45eb-a88e-cac0bb987439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:05.497538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 185/363] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:05.498475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1187efa-84dc-45eb-a88e-cac0bb987439 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:05.516977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:05.538132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 179/364] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:05.539081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7f9c3b-c626-4172-972e-077bdc381d85 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:05.626789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1187efa-84dc-45eb-a88e-cac0bb987439 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:05.645715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:05.664185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7f9c3b-c626-4172-972e-077bdc381d85 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:05.675332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32567ffd-6c45-43d3-9d5c-074844ebee86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:05.675977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32567ffd-6c45-43d3-9d5c-074844ebee86 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:05.676268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32567ffd-6c45-43d3-9d5c-074844ebee86 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:05.676551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32567ffd-6c45-43d3-9d5c-074844ebee86 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:05.677314 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32567ffd-6c45-43d3-9d5c-074844ebee86 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:05.677627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32567ffd-6c45-43d3-9d5c-074844ebee86 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:05.679022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32567ffd-6c45-43d3-9d5c-074844ebee86 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:05.692572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 186/365] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:05.693538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32567ffd-6c45-43d3-9d5c-074844ebee86 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:05.708236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d567b9-71a5-43f3-a288-3f243179e688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:05.708925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d567b9-71a5-43f3-a288-3f243179e688 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:05.709217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d567b9-71a5-43f3-a288-3f243179e688 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:05.709503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d567b9-71a5-43f3-a288-3f243179e688 None admin] PATH_INFO: `/v3/users/8f220d426c4a4f1b89dcd29589a5be4c` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:05.710760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d567b9-71a5-43f3-a288-3f243179e688 None admin] RBAC: Authorizing `identity:get_user(user_id=8f220d426c4a4f1b89dcd29589a5be4c)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:05.712288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d567b9-71a5-43f3-a288-3f243179e688 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:05.714112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 180/366] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:34:05 2022] GET /identity/v3/users/8f220d426c4a4f1b89dcd29589a5be4c => generated 293 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:05.714931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d567b9-71a5-43f3-a288-3f243179e688 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:05.801034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:05.809097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32567ffd-6c45-43d3-9d5c-074844ebee86 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:05.837059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d567b9-71a5-43f3-a288-3f243179e688 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:05.847078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f396a54e-69e7-4f2a-a061-180d75cbe341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:05.847698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f396a54e-69e7-4f2a-a061-180d75cbe341 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:05.847991 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f396a54e-69e7-4f2a-a061-180d75cbe341 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:05.848273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f396a54e-69e7-4f2a-a061-180d75cbe341 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:05.860102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f396a54e-69e7-4f2a-a061-180d75cbe341 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-f396a54e-69e7-4f2a-a061-180d75cbe341 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:05.866380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:05.867291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:05.868105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 187/367] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:05.868818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f396a54e-69e7-4f2a-a061-180d75cbe341 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:05.875364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73465e50-80b2-47a2-b772-5edaa3d906e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:05.876077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73465e50-80b2-47a2-b772-5edaa3d906e3 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:05.876373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73465e50-80b2-47a2-b772-5edaa3d906e3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:05.876711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73465e50-80b2-47a2-b772-5edaa3d906e3 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:05.878173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73465e50-80b2-47a2-b772-5edaa3d906e3 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:05.880273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73465e50-80b2-47a2-b772-5edaa3d906e3 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:05.882398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 181/368] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:05 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:05.883246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73465e50-80b2-47a2-b772-5edaa3d906e3 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:05.916476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:05.982867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f396a54e-69e7-4f2a-a061-180d75cbe341 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:05.994205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73465e50-80b2-47a2-b772-5edaa3d906e3 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:06.021476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:06.028414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9984c5-3467-4d53-b48d-38c13722d2dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:06.029251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9984c5-3467-4d53-b48d-38c13722d2dc None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:06.029546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9984c5-3467-4d53-b48d-38c13722d2dc None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:06.030198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9984c5-3467-4d53-b48d-38c13722d2dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:06.036012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb36471-8add-4ec1-8868-21202cf8341c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:06.036664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb36471-8add-4ec1-8868-21202cf8341c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:06.036968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb36471-8add-4ec1-8868-21202cf8341c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:06.037290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb36471-8add-4ec1-8868-21202cf8341c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:06.038087 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb36471-8add-4ec1-8868-21202cf8341c None admin] RBAC: Adding query filter params (role.id=f382f86b952e4b869017bbf06295b028, user.id=8f220d426c4a4f1b89dcd29589a5be4c, scope.system=all) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:06.038402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb36471-8add-4ec1-8868-21202cf8341c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:06.039844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb36471-8add-4ec1-8868-21202cf8341c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:06.059614 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 182/369] 10.222.0.40 () {56 vars in 1308 bytes} [Fri Jul 29 04:34:05 2022] GET /identity/v3/role_assignments?user.id=8f220d426c4a4f1b89dcd29589a5be4c&scope.system=all&role.id=f382f86b952e4b869017bbf06295b028 => generated 510 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:06.060543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb36471-8add-4ec1-8868-21202cf8341c None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:06.134124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9984c5-3467-4d53-b48d-38c13722d2dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:06.136170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9984c5-3467-4d53-b48d-38c13722d2dc None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:06.218651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb36471-8add-4ec1-8868-21202cf8341c None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:06.238448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:06.270517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7428da-9ff8-44d5-9ef4-bae27c2c5ae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:06.271412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7428da-9ff8-44d5-9ef4-bae27c2c5ae0 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:06.271870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7428da-9ff8-44d5-9ef4-bae27c2c5ae0 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:06.272330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7428da-9ff8-44d5-9ef4-bae27c2c5ae0 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:06.273256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7428da-9ff8-44d5-9ef4-bae27c2c5ae0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:06.273722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7428da-9ff8-44d5-9ef4-bae27c2c5ae0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:06.275324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d7428da-9ff8-44d5-9ef4-bae27c2c5ae0 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:06.293339 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 183/370] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:06.294464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7428da-9ff8-44d5-9ef4-bae27c2c5ae0 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:06.357980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:06.361036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 188/371] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 493 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:06.362668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9984c5-3467-4d53-b48d-38c13722d2dc None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:06.434275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7428da-9ff8-44d5-9ef4-bae27c2c5ae0 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:06.477306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7bd284-304a-4c49-b73e-62568ecfac74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:06.478016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7bd284-304a-4c49-b73e-62568ecfac74 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:06.478308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7bd284-304a-4c49-b73e-62568ecfac74 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:06.478593 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7bd284-304a-4c49-b73e-62568ecfac74 None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:06.479805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7bd284-304a-4c49-b73e-62568ecfac74 None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:06.481325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7bd284-304a-4c49-b73e-62568ecfac74 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:06.483256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 184/372] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:06 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:06.484117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7bd284-304a-4c49-b73e-62568ecfac74 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:06.500335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9984c5-3467-4d53-b48d-38c13722d2dc None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:06.545050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b669fd9-d19e-4642-8797-45557f0eeca1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:06.546114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b669fd9-d19e-4642-8797-45557f0eeca1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:06.546406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b669fd9-d19e-4642-8797-45557f0eeca1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:06.546969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b669fd9-d19e-4642-8797-45557f0eeca1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:06.563786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:06.624823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7bd284-304a-4c49-b73e-62568ecfac74 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:06.637743 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:06.645423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b669fd9-d19e-4642-8797-45557f0eeca1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:06.647097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b669fd9-d19e-4642-8797-45557f0eeca1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:06.683784 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c911676-7e8c-4e22-9f0d-12cded51cd5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:06.684686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c911676-7e8c-4e22-9f0d-12cded51cd5e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:06.685236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c911676-7e8c-4e22-9f0d-12cded51cd5e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:06.685536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c911676-7e8c-4e22-9f0d-12cded51cd5e None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:06.686998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c911676-7e8c-4e22-9f0d-12cded51cd5e None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:06.688944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c911676-7e8c-4e22-9f0d-12cded51cd5e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:06.692234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 185/373] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:06 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:06.693411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c911676-7e8c-4e22-9f0d-12cded51cd5e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:06.775944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 189/374] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:06.777561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b669fd9-d19e-4642-8797-45557f0eeca1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:06.832304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:06.837924 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c911676-7e8c-4e22-9f0d-12cded51cd5e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:06.885303 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cf7695-492a-4369-8df4-4687f95826b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:06.885946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cf7695-492a-4369-8df4-4687f95826b4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:06.886231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cf7695-492a-4369-8df4-4687f95826b4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:06.886508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cf7695-492a-4369-8df4-4687f95826b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:06.887391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cf7695-492a-4369-8df4-4687f95826b4 None admin] RBAC: Adding query filter params (group.id=9f588151b280456981cfc088281fb10b, role.id=9e77a799dd224c1480f51a54f3aed529, scope.project.id=7a2a97f3a55842ac8262f51748e570c2) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:06.887711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cf7695-492a-4369-8df4-4687f95826b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:06.889336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cf7695-492a-4369-8df4-4687f95826b4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:06.896949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 186/375] 10.222.0.40 () {56 vars in 1376 bytes} [Fri Jul 29 04:34:06 2022] GET /identity/v3/role_assignments?group.id=9f588151b280456981cfc088281fb10b&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 260 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:06.897900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cf7695-492a-4369-8df4-4687f95826b4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:06.915908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b669fd9-d19e-4642-8797-45557f0eeca1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:06.953606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:06.976258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ab078e-97a6-4617-8f8d-b589e0b6bac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:06.977467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ab078e-97a6-4617-8f8d-b589e0b6bac3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:06.977770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ab078e-97a6-4617-8f8d-b589e0b6bac3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:06.978064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ab078e-97a6-4617-8f8d-b589e0b6bac3 None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:06.982357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ab078e-97a6-4617-8f8d-b589e0b6bac3 None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:06.984406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ab078e-97a6-4617-8f8d-b589e0b6bac3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:06.991274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 190/376] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:06 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:06.991274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53813836-f6f6-4a4f-b806-b35139522482 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:06.991274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53813836-f6f6-4a4f-b806-b35139522482 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:06.991274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53813836-f6f6-4a4f-b806-b35139522482 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:06.991274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 191/377] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:06.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78712475-ff6a-494e-b8fd-cce4592c8afe None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:06.997978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78712475-ff6a-494e-b8fd-cce4592c8afe None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:06.998255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78712475-ff6a-494e-b8fd-cce4592c8afe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.024369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-78712475-ff6a-494e-b8fd-cce4592c8afe None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:07.026474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-78712475-ff6a-494e-b8fd-cce4592c8afe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:07.041414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cf7695-492a-4369-8df4-4687f95826b4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:07.086695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b369df83-63f3-4e4e-8567-4023be9af038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:07.087883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b369df83-63f3-4e4e-8567-4023be9af038 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:07.088453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b369df83-63f3-4e4e-8567-4023be9af038 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:07.088993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b369df83-63f3-4e4e-8567-4023be9af038 None admin] PATH_INFO: `/v3/projects/25a26196f3ac436c876fd26eb5482e4a` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.090714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b369df83-63f3-4e4e-8567-4023be9af038 None admin] RBAC: Authorizing `identity:get_project(project_id=25a26196f3ac436c876fd26eb5482e4a)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:07.092658 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b369df83-63f3-4e4e-8567-4023be9af038 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:07.095096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 187/378] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:06 2022] GET /identity/v3/projects/25a26196f3ac436c876fd26eb5482e4a => generated 344 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:07.096796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b369df83-63f3-4e4e-8567-4023be9af038 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:07.121697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 192/379] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:07.122362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78712475-ff6a-494e-b8fd-cce4592c8afe None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:07.185903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:07.240411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b369df83-63f3-4e4e-8567-4023be9af038 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:07.253553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78712475-ff6a-494e-b8fd-cce4592c8afe None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:07.255119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:07.281350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961239cd-b578-4b10-8ed8-b0d3283d4dd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:07.282109 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961239cd-b578-4b10-8ed8-b0d3283d4dd3 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:07.282405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961239cd-b578-4b10-8ed8-b0d3283d4dd3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:07.282747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961239cd-b578-4b10-8ed8-b0d3283d4dd3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.283791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961239cd-b578-4b10-8ed8-b0d3283d4dd3 None admin] RBAC: Adding query filter params (group.id=9f588151b280456981cfc088281fb10b, role.id=28ddf4e7cb7f4139a3b6a79077418921, scope.project.id=7a2a97f3a55842ac8262f51748e570c2) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:07.284219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961239cd-b578-4b10-8ed8-b0d3283d4dd3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:07.285813 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961239cd-b578-4b10-8ed8-b0d3283d4dd3 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:07.289793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc7646d-ae78-4bc1-ab80-a756e4e9ddf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:07.290526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc7646d-ae78-4bc1-ab80-a756e4e9ddf5 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:07.290888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc7646d-ae78-4bc1-ab80-a756e4e9ddf5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:07.291239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc7646d-ae78-4bc1-ab80-a756e4e9ddf5 None admin] PATH_INFO: `/v3/groups/316757d7a0b34e3d94f7c18a30d1a3fa` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.294312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 188/380] 10.222.0.40 () {56 vars in 1376 bytes} [Fri Jul 29 04:34:07 2022] GET /identity/v3/role_assignments?group.id=9f588151b280456981cfc088281fb10b&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=28ddf4e7cb7f4139a3b6a79077418921 => generated 260 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:07.295523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961239cd-b578-4b10-8ed8-b0d3283d4dd3 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:07.298209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc7646d-ae78-4bc1-ab80-a756e4e9ddf5 None admin] RBAC: Authorizing `identity:get_group(group_id=316757d7a0b34e3d94f7c18a30d1a3fa)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:07.301783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc7646d-ae78-4bc1-ab80-a756e4e9ddf5 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:07.303080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 193/381] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:07 2022] GET /identity/v3/groups/316757d7a0b34e3d94f7c18a30d1a3fa => generated 229 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:07.313350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc7646d-ae78-4bc1-ab80-a756e4e9ddf5 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:07.489388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc7646d-ae78-4bc1-ab80-a756e4e9ddf5 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:07.491290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:07.496994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961239cd-b578-4b10-8ed8-b0d3283d4dd3 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:07.527006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e474affc-5ef3-4de7-a311-550e72f0df5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:07.527812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e474affc-5ef3-4de7-a311-550e72f0df5a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:07.528105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e474affc-5ef3-4de7-a311-550e72f0df5a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:07.528385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e474affc-5ef3-4de7-a311-550e72f0df5a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.529229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e474affc-5ef3-4de7-a311-550e72f0df5a None admin] RBAC: Adding query filter params (group.id=316757d7a0b34e3d94f7c18a30d1a3fa, role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=25a26196f3ac436c876fd26eb5482e4a) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:07.529543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e474affc-5ef3-4de7-a311-550e72f0df5a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:07.530996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e474affc-5ef3-4de7-a311-550e72f0df5a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:07.537153 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 194/382] 10.222.0.40 () {56 vars in 1376 bytes} [Fri Jul 29 04:34:07 2022] GET /identity/v3/role_assignments?group.id=316757d7a0b34e3d94f7c18a30d1a3fa&scope.project.id=25a26196f3ac436c876fd26eb5482e4a&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 260 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:07.545755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e42ca88-eddd-409c-8ffd-d38edd7b395b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:07.546442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e42ca88-eddd-409c-8ffd-d38edd7b395b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:07.546733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e42ca88-eddd-409c-8ffd-d38edd7b395b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:07.547063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e42ca88-eddd-409c-8ffd-d38edd7b395b None admin] PATH_INFO: `/v3/roles/member` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.561544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e42ca88-eddd-409c-8ffd-d38edd7b395b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-4e42ca88-eddd-409c-8ffd-d38edd7b395b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:07.567837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:07.569026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:07.569807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 189/383] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:07.574919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cac7bc1-4a84-4680-ba1f-44ed3a57e9d8 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:07.575212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cac7bc1-4a84-4680-ba1f-44ed3a57e9d8 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:07.575494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cac7bc1-4a84-4680-ba1f-44ed3a57e9d8 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.577210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 195/384] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:07.578875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e42ca88-eddd-409c-8ffd-d38edd7b395b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:07.583311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e7cdf1-a522-49e5-be92-76c22edc9d5a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:07.583607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e7cdf1-a522-49e5-be92-76c22edc9d5a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:07.583887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e7cdf1-a522-49e5-be92-76c22edc9d5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.605055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-47e7cdf1-a522-49e5-be92-76c22edc9d5a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:07.606285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-47e7cdf1-a522-49e5-be92-76c22edc9d5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:07.626610 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:07.694336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 196/385] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:34:07.699480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e42ca88-eddd-409c-8ffd-d38edd7b395b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:07.721819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e7cdf1-a522-49e5-be92-76c22edc9d5a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:07.740007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b99635f-6083-4eba-ac58-f38f7992b352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:07.741258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b99635f-6083-4eba-ac58-f38f7992b352 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:07.742242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b99635f-6083-4eba-ac58-f38f7992b352 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:07.742884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b99635f-6083-4eba-ac58-f38f7992b352 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.743909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b99635f-6083-4eba-ac58-f38f7992b352 None admin] RBAC: Adding query filter params (name=member) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:07.744567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b99635f-6083-4eba-ac58-f38f7992b352 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:07.746572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b99635f-6083-4eba-ac58-f38f7992b352 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:07.760911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 190/386] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:34:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:07.769660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b99635f-6083-4eba-ac58-f38f7992b352 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:07.879750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:07.889605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e7cdf1-a522-49e5-be92-76c22edc9d5a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:07.918770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:07.928362 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b99635f-6083-4eba-ac58-f38f7992b352 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:07.968221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7b7a80-8aa7-46f7-aa53-5b26ac9ad952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:07.968904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7b7a80-8aa7-46f7-aa53-5b26ac9ad952 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:07.969186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7b7a80-8aa7-46f7-aa53-5b26ac9ad952 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:07.969460 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7b7a80-8aa7-46f7-aa53-5b26ac9ad952 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.986092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7b7a80-8aa7-46f7-aa53-5b26ac9ad952 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-3a7b7a80-8aa7-46f7-aa53-5b26ac9ad952 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:07.992768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:07.993747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:07.994520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:07.995743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 197/387] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:34:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:07.996779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12c63d7-2049-41a7-a6f8-6ed353f9d629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:07.997440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12c63d7-2049-41a7-a6f8-6ed353f9d629 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:07.997724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12c63d7-2049-41a7-a6f8-6ed353f9d629 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:07.998002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12c63d7-2049-41a7-a6f8-6ed353f9d629 None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:07.999169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12c63d7-2049-41a7-a6f8-6ed353f9d629 None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:08.000637 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12c63d7-2049-41a7-a6f8-6ed353f9d629 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:08.002341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 191/388] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:07 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:08.003271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7b7a80-8aa7-46f7-aa53-5b26ac9ad952 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:08.008288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12c63d7-2049-41a7-a6f8-6ed353f9d629 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:08.184643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12c63d7-2049-41a7-a6f8-6ed353f9d629 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:08.188314 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7b7a80-8aa7-46f7-aa53-5b26ac9ad952 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:08.220377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae56503-29ae-42ba-a294-0f3b66b04b20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:08.221607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae56503-29ae-42ba-a294-0f3b66b04b20 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:08.221899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae56503-29ae-42ba-a294-0f3b66b04b20 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:08.222186 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae56503-29ae-42ba-a294-0f3b66b04b20 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:08.223394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae56503-29ae-42ba-a294-0f3b66b04b20 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:08.225958 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae56503-29ae-42ba-a294-0f3b66b04b20 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:08.227969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 192/389] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:08 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:08.233423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae56503-29ae-42ba-a294-0f3b66b04b20 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:08.237641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47f8957-c7c4-4133-8cbe-9dc930b3daf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:08.239213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47f8957-c7c4-4133-8cbe-9dc930b3daf0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:08.239500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47f8957-c7c4-4133-8cbe-9dc930b3daf0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:08.239779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47f8957-c7c4-4133-8cbe-9dc930b3daf0 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:08.240511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47f8957-c7c4-4133-8cbe-9dc930b3daf0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:08.240846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47f8957-c7c4-4133-8cbe-9dc930b3daf0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:08.242227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47f8957-c7c4-4133-8cbe-9dc930b3daf0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:08.260762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 198/390] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:08.268214 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47f8957-c7c4-4133-8cbe-9dc930b3daf0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:08.297061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:08.305507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:08.393288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae56503-29ae-42ba-a294-0f3b66b04b20 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:08.453749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47f8957-c7c4-4133-8cbe-9dc930b3daf0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:08.474068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0dfe47-904a-4485-b676-88c69785a8d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:08.474766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0dfe47-904a-4485-b676-88c69785a8d5 None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:08.475054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0dfe47-904a-4485-b676-88c69785a8d5 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:08.475335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0dfe47-904a-4485-b676-88c69785a8d5 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/groups/9f588151b280456981cfc088281fb10b/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:08.479062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0dfe47-904a-4485-b676-88c69785a8d5 None admin] RBAC: Authorizing `identity:create_grant(project_id=e96107c2d25740e783b49bd3a96cbc8d, group_id=9f588151b280456981cfc088281fb10b, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:08.480893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0dfe47-904a-4485-b676-88c69785a8d5 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:08.493782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21a5184-701a-4537-8ce6-72e1c4900295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:08.494566 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21a5184-701a-4537-8ce6-72e1c4900295 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:08.494977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21a5184-701a-4537-8ce6-72e1c4900295 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:08.495235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21a5184-701a-4537-8ce6-72e1c4900295 None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:08.496752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21a5184-701a-4537-8ce6-72e1c4900295 None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:08.498479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21a5184-701a-4537-8ce6-72e1c4900295 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:08.500375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 199/391] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:08 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:08.503821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 193/392] 10.222.0.40 () {58 vars in 1442 bytes} [Fri Jul 29 04:34:08 2022] PUT /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/groups/9f588151b280456981cfc088281fb10b/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:34:08.508714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0dfe47-904a-4485-b676-88c69785a8d5 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:08.627197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:08.638929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0dfe47-904a-4485-b676-88c69785a8d5 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:08.687042 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b5faec-1cc8-4df9-9e20-69541bd606db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:08.687716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5faec-1cc8-4df9-9e20-69541bd606db None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:08.688004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5faec-1cc8-4df9-9e20-69541bd606db None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:08.688288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b5faec-1cc8-4df9-9e20-69541bd606db None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:08.693486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5faec-1cc8-4df9-9e20-69541bd606db None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:08.694995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b5faec-1cc8-4df9-9e20-69541bd606db None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:08.697186 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 194/393] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:08 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:08.702971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21a5184-701a-4537-8ce6-72e1c4900295 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:08.796978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:08.812895 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21a5184-701a-4537-8ce6-72e1c4900295 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:08.849795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58498e25-9df0-485a-836e-7563c6c66eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:08.850497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58498e25-9df0-485a-836e-7563c6c66eb4 None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:08.850796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58498e25-9df0-485a-836e-7563c6c66eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:08.851083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58498e25-9df0-485a-836e-7563c6c66eb4 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/groups/9f588151b280456981cfc088281fb10b/roles/28ddf4e7cb7f4139a3b6a79077418921` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:08.853497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58498e25-9df0-485a-836e-7563c6c66eb4 None admin] RBAC: Authorizing `identity:create_grant(project_id=e96107c2d25740e783b49bd3a96cbc8d, group_id=9f588151b280456981cfc088281fb10b, role_id=28ddf4e7cb7f4139a3b6a79077418921)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:08.855016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58498e25-9df0-485a-836e-7563c6c66eb4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:08.876887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 200/394] 10.222.0.40 () {58 vars in 1442 bytes} [Fri Jul 29 04:34:08 2022] PUT /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d/groups/9f588151b280456981cfc088281fb10b/roles/28ddf4e7cb7f4139a3b6a79077418921 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:09.580716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4dad78-5e42-4d7c-831b-af9eb3090b6a None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:09.581227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4dad78-5e42-4d7c-831b-af9eb3090b6a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:09.581471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4dad78-5e42-4d7c-831b-af9eb3090b6a None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:09.583134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 195/395] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:09.587984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a444a7bf-88e5-46ba-bd2f-326b6324bdd7 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:09.588283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a444a7bf-88e5-46ba-bd2f-326b6324bdd7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:09.588567 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a444a7bf-88e5-46ba-bd2f-326b6324bdd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:09.617534 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a444a7bf-88e5-46ba-bd2f-326b6324bdd7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:09.618755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a444a7bf-88e5-46ba-bd2f-326b6324bdd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:09.701493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 201/396] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:09.733316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4dad78-5e42-4d7c-831b-af9eb3090b6a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:09.857380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4dad78-5e42-4d7c-831b-af9eb3090b6a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:09.903587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dc9f8e-9885-489a-8c4e-25acad2af4a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:09.904241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dc9f8e-9885-489a-8c4e-25acad2af4a3 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:09.904524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dc9f8e-9885-489a-8c4e-25acad2af4a3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:09.904827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dc9f8e-9885-489a-8c4e-25acad2af4a3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:09.920424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03dc9f8e-9885-489a-8c4e-25acad2af4a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-03dc9f8e-9885-489a-8c4e-25acad2af4a3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:09.926248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:09.927129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:09.927906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 196/397] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:09.934875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a444a7bf-88e5-46ba-bd2f-326b6324bdd7 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:09.988645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb293efa-2eaa-4334-a52a-c50fc0a7aa32 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:09.988945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb293efa-2eaa-4334-a52a-c50fc0a7aa32 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:09.989205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb293efa-2eaa-4334-a52a-c50fc0a7aa32 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:09.990798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 197/398] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:09.996039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a959a7-9322-4004-87fb-b3283bd98bee None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:09.996324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a959a7-9322-4004-87fb-b3283bd98bee None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:09.996619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a959a7-9322-4004-87fb-b3283bd98bee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:10.034528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-54a959a7-9322-4004-87fb-b3283bd98bee None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:10.035752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-54a959a7-9322-4004-87fb-b3283bd98bee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:10.041819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:10.059066 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:10.089224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a444a7bf-88e5-46ba-bd2f-326b6324bdd7 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:10.158824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 198/399] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:34:10.168060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4117bd9-707a-43eb-9b1d-c6d8d6b1cadf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:10.168719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4117bd9-707a-43eb-9b1d-c6d8d6b1cadf None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:10.169014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4117bd9-707a-43eb-9b1d-c6d8d6b1cadf None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:10.169298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4117bd9-707a-43eb-9b1d-c6d8d6b1cadf None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:10.170035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4117bd9-707a-43eb-9b1d-c6d8d6b1cadf None admin] RBAC: Adding query filter params (name=member) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:10.170347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4117bd9-707a-43eb-9b1d-c6d8d6b1cadf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:10.171752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4117bd9-707a-43eb-9b1d-c6d8d6b1cadf None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:10.183354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a959a7-9322-4004-87fb-b3283bd98bee None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:10.203395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 202/400] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:34:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:10.207889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4117bd9-707a-43eb-9b1d-c6d8d6b1cadf None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:10.324639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4117bd9-707a-43eb-9b1d-c6d8d6b1cadf None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:10.372756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12181bc2-7949-41a2-b147-4c3b32b415e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:10.373409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12181bc2-7949-41a2-b147-4c3b32b415e9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:10.373699 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12181bc2-7949-41a2-b147-4c3b32b415e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:10.373980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12181bc2-7949-41a2-b147-4c3b32b415e9 None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:10.376365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12181bc2-7949-41a2-b147-4c3b32b415e9 None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:10.378062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12181bc2-7949-41a2-b147-4c3b32b415e9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:10.380981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 203/401] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:10 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:10.389716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12181bc2-7949-41a2-b147-4c3b32b415e9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:10.402071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:10.413589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a959a7-9322-4004-87fb-b3283bd98bee None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:10.473389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48a4069-003a-49d1-bc84-0e6ef84366b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:10.474056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48a4069-003a-49d1-bc84-0e6ef84366b8 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:10.474343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48a4069-003a-49d1-bc84-0e6ef84366b8 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:10.474623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48a4069-003a-49d1-bc84-0e6ef84366b8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:10.491411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48a4069-003a-49d1-bc84-0e6ef84366b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-b48a4069-003a-49d1-bc84-0e6ef84366b8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:10.505324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:10.506263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:10.507044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 199/402] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:34:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:10.512296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48a4069-003a-49d1-bc84-0e6ef84366b8 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:10.516523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12181bc2-7949-41a2-b147-4c3b32b415e9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:10.532273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:10.558497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2440453c-b524-402f-b726-45f71c73a911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:10.559305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2440453c-b524-402f-b726-45f71c73a911 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:10.560487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2440453c-b524-402f-b726-45f71c73a911 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:10.560909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2440453c-b524-402f-b726-45f71c73a911 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:10.562225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2440453c-b524-402f-b726-45f71c73a911 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:10.563790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2440453c-b524-402f-b726-45f71c73a911 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:10.570916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 204/403] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:10 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:10.570916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce4e43-1aec-4d06-9cd3-c3a290879483 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:10.570916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce4e43-1aec-4d06-9cd3-c3a290879483 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:10.570916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ce4e43-1aec-4d06-9cd3-c3a290879483 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:10.570916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 205/404] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:10.573573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce4e43-1aec-4d06-9cd3-c3a290879483 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:10.643120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48a4069-003a-49d1-bc84-0e6ef84366b8 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:10.683131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d407251c-5b0c-4ff8-8604-bf829b98a9fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:10.683845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d407251c-5b0c-4ff8-8604-bf829b98a9fb None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:10.684211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d407251c-5b0c-4ff8-8604-bf829b98a9fb None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:10.684570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d407251c-5b0c-4ff8-8604-bf829b98a9fb None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:10.685404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d407251c-5b0c-4ff8-8604-bf829b98a9fb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:10.685794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d407251c-5b0c-4ff8-8604-bf829b98a9fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:10.686944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:10.687385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d407251c-5b0c-4ff8-8604-bf829b98a9fb None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:10.697840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 200/405] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:10.699191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9502f41b-9cfa-43f7-b6e8-c673bcc5f729 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:10.699555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9502f41b-9cfa-43f7-b6e8-c673bcc5f729 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:10.699931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9502f41b-9cfa-43f7-b6e8-c673bcc5f729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:10.718809 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-9502f41b-9cfa-43f7-b6e8-c673bcc5f729 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:10.720052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ce4e43-1aec-4d06-9cd3-c3a290879483 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:10.720244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9502f41b-9cfa-43f7-b6e8-c673bcc5f729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:10.766238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7e8037-7846-4443-be4b-5b614fa66dc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:10.766950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7e8037-7846-4443-be4b-5b614fa66dc5 None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:10.768307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7e8037-7846-4443-be4b-5b614fa66dc5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:10.768672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7e8037-7846-4443-be4b-5b614fa66dc5 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2/groups/9f588151b280456981cfc088281fb10b/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:10.771354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7e8037-7846-4443-be4b-5b614fa66dc5 None admin] RBAC: Authorizing `identity:create_grant(project_id=7a2a97f3a55842ac8262f51748e570c2, group_id=9f588151b280456981cfc088281fb10b, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:10.773095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7e8037-7846-4443-be4b-5b614fa66dc5 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:10.792406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 206/406] 10.222.0.40 () {58 vars in 1442 bytes} [Fri Jul 29 04:34:10 2022] PUT /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2/groups/9f588151b280456981cfc088281fb10b/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:10.793609 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7e8037-7846-4443-be4b-5b614fa66dc5 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:10.808766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 201/407] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:10.823938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9502f41b-9cfa-43f7-b6e8-c673bcc5f729 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:10.826686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:10.914528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7e8037-7846-4443-be4b-5b614fa66dc5 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:10.954696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0715a7-4b7b-48b3-9b18-8548d2e3561d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:10.957504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0715a7-4b7b-48b3-9b18-8548d2e3561d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:10.957504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0715a7-4b7b-48b3-9b18-8548d2e3561d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:10.957504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0715a7-4b7b-48b3-9b18-8548d2e3561d None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:10.958396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0715a7-4b7b-48b3-9b18-8548d2e3561d None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:10.960071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0715a7-4b7b-48b3-9b18-8548d2e3561d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:10.962276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 207/408] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:10 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:10.966181 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9502f41b-9cfa-43f7-b6e8-c673bcc5f729 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:10.967812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0715a7-4b7b-48b3-9b18-8548d2e3561d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:10.971146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:11.021169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43d8867-4c6b-43f9-9ff6-fc6c97313f90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:11.021832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43d8867-4c6b-43f9-9ff6-fc6c97313f90 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.022110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43d8867-4c6b-43f9-9ff6-fc6c97313f90 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.022383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43d8867-4c6b-43f9-9ff6-fc6c97313f90 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.039059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43d8867-4c6b-43f9-9ff6-fc6c97313f90 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-a43d8867-4c6b-43f9-9ff6-fc6c97313f90 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:11.046885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:11.048060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:11.049675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 202/409] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:11.056705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43d8867-4c6b-43f9-9ff6-fc6c97313f90 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:11.135492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0715a7-4b7b-48b3-9b18-8548d2e3561d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:11.162797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:11.205797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5adb01d1-fba6-4ac5-b528-544f8927560a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:11.206597 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adb01d1-fba6-4ac5-b528-544f8927560a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.206888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adb01d1-fba6-4ac5-b528-544f8927560a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.207169 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5adb01d1-fba6-4ac5-b528-544f8927560a None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.208473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5adb01d1-fba6-4ac5-b528-544f8927560a None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:11.211365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5adb01d1-fba6-4ac5-b528-544f8927560a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:11.213252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 208/410] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:10 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:11.215115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9691f213-016d-487d-9522-eaa3c4f4b8b6 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.215401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9691f213-016d-487d-9522-eaa3c4f4b8b6 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.215681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9691f213-016d-487d-9522-eaa3c4f4b8b6 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.217300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 209/411] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:34:11.222096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9691f213-016d-487d-9522-eaa3c4f4b8b6 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:11.224424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43d8867-4c6b-43f9-9ff6-fc6c97313f90 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:11.261257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb8d455-82f7-425b-8e84-732df7e911fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:11.261980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb8d455-82f7-425b-8e84-732df7e911fe None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.262455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb8d455-82f7-425b-8e84-732df7e911fe None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.262808 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb8d455-82f7-425b-8e84-732df7e911fe None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.263744 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb8d455-82f7-425b-8e84-732df7e911fe None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:11.264121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb8d455-82f7-425b-8e84-732df7e911fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:11.265604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb8d455-82f7-425b-8e84-732df7e911fe None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:11.274697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 203/412] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:11.276041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f1804b-030e-4648-86ff-d7dcf1a06df0 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.276460 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f1804b-030e-4648-86ff-d7dcf1a06df0 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.276838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f1804b-030e-4648-86ff-d7dcf1a06df0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.295387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-85f1804b-030e-4648-86ff-d7dcf1a06df0 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:11.296587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-85f1804b-030e-4648-86ff-d7dcf1a06df0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:11.302167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:11.340269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9691f213-016d-487d-9522-eaa3c4f4b8b6 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:11.369621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 204/413] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:11.371185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfbd483-dd70-46db-a156-7254db3019ef None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.371681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfbd483-dd70-46db-a156-7254db3019ef None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.372157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfbd483-dd70-46db-a156-7254db3019ef None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.373667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 205/414] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:11.374551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfbd483-dd70-46db-a156-7254db3019ef None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:11.394493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41bdea3-025c-4913-8d1c-9ef8e75cc5ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:11.395182 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41bdea3-025c-4913-8d1c-9ef8e75cc5ba None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.395494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41bdea3-025c-4913-8d1c-9ef8e75cc5ba None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.395773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41bdea3-025c-4913-8d1c-9ef8e75cc5ba None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2/groups/9f588151b280456981cfc088281fb10b/roles/28ddf4e7cb7f4139a3b6a79077418921` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.398145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41bdea3-025c-4913-8d1c-9ef8e75cc5ba None admin] RBAC: Authorizing `identity:create_grant(project_id=7a2a97f3a55842ac8262f51748e570c2, group_id=9f588151b280456981cfc088281fb10b, role_id=28ddf4e7cb7f4139a3b6a79077418921)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:11.399600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41bdea3-025c-4913-8d1c-9ef8e75cc5ba None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:11.422140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 210/415] 10.222.0.40 () {58 vars in 1442 bytes} [Fri Jul 29 04:34:11 2022] PUT /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2/groups/9f588151b280456981cfc088281fb10b/roles/28ddf4e7cb7f4139a3b6a79077418921 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:11.422140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b585e4-662e-4409-8ae3-a39e1324bbe1 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.422140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b585e4-662e-4409-8ae3-a39e1324bbe1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.422140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b585e4-662e-4409-8ae3-a39e1324bbe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.438881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d9b585e4-662e-4409-8ae3-a39e1324bbe1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:11.440243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d9b585e4-662e-4409-8ae3-a39e1324bbe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:11.461602 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:11.518538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfbd483-dd70-46db-a156-7254db3019ef None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:11.536223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 211/416] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:11.537242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b585e4-662e-4409-8ae3-a39e1324bbe1 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:11.582098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825c77e0-2a0d-4d2c-8fa0-6e46251f3bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:11.582760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c77e0-2a0d-4d2c-8fa0-6e46251f3bfa None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.583115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c77e0-2a0d-4d2c-8fa0-6e46251f3bfa None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.583394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825c77e0-2a0d-4d2c-8fa0-6e46251f3bfa None admin] PATH_INFO: `/v3/groups/316757d7a0b34e3d94f7c18a30d1a3fa` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.584676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825c77e0-2a0d-4d2c-8fa0-6e46251f3bfa None admin] RBAC: Authorizing `identity:get_group(group_id=316757d7a0b34e3d94f7c18a30d1a3fa)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:11.586084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825c77e0-2a0d-4d2c-8fa0-6e46251f3bfa None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:11.587662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 206/417] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:11 2022] GET /identity/v3/groups/316757d7a0b34e3d94f7c18a30d1a3fa => generated 229 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:11.588577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825c77e0-2a0d-4d2c-8fa0-6e46251f3bfa None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:11.619401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:11.673092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b585e4-662e-4409-8ae3-a39e1324bbe1 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:11.724816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d28ac0-92fb-4bb7-88f9-124a052e0d59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:11.725574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825c77e0-2a0d-4d2c-8fa0-6e46251f3bfa None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:11.726031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d28ac0-92fb-4bb7-88f9-124a052e0d59 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.726031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d28ac0-92fb-4bb7-88f9-124a052e0d59 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.726370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d28ac0-92fb-4bb7-88f9-124a052e0d59 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.741707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d28ac0-92fb-4bb7-88f9-124a052e0d59 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-f4d28ac0-92fb-4bb7-88f9-124a052e0d59 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:11.746711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:11.747628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:11.748406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 212/418] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:11.750116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d28ac0-92fb-4bb7-88f9-124a052e0d59 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:11.755099 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:11.770045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e544a781-659d-4a15-aba4-476bb0dff549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:11.770893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e544a781-659d-4a15-aba4-476bb0dff549 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.771263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e544a781-659d-4a15-aba4-476bb0dff549 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.771632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e544a781-659d-4a15-aba4-476bb0dff549 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.783326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e544a781-659d-4a15-aba4-476bb0dff549 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-e544a781-659d-4a15-aba4-476bb0dff549 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:11.789062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:11.790886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:11.792483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 207/419] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:34:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:11.800637 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e544a781-659d-4a15-aba4-476bb0dff549 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:11.860435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d28ac0-92fb-4bb7-88f9-124a052e0d59 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:11.882483 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:11.897519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f08016f-b1d6-4b55-9ab1-d5c2925691d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:11.898398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f08016f-b1d6-4b55-9ab1-d5c2925691d4 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.898773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f08016f-b1d6-4b55-9ab1-d5c2925691d4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.899120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f08016f-b1d6-4b55-9ab1-d5c2925691d4 None admin] PATH_INFO: `/v3/projects/25a26196f3ac436c876fd26eb5482e4a` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.900426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f08016f-b1d6-4b55-9ab1-d5c2925691d4 None admin] RBAC: Authorizing `identity:get_project(project_id=25a26196f3ac436c876fd26eb5482e4a)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:11.902171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f08016f-b1d6-4b55-9ab1-d5c2925691d4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:11.909066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 213/420] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:11 2022] GET /identity/v3/projects/25a26196f3ac436c876fd26eb5482e4a => generated 344 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:11.910010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f08016f-b1d6-4b55-9ab1-d5c2925691d4 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:11.911085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e544a781-659d-4a15-aba4-476bb0dff549 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:11.949232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e01e231-72a3-43d4-a179-197abca7b738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:11.950722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e01e231-72a3-43d4-a179-197abca7b738 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:11.950722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e01e231-72a3-43d4-a179-197abca7b738 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:11.950722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e01e231-72a3-43d4-a179-197abca7b738 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:11.951161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e01e231-72a3-43d4-a179-197abca7b738 None admin] RBAC: Adding query filter params (name=member) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:11.953540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e01e231-72a3-43d4-a179-197abca7b738 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:11.953540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e01e231-72a3-43d4-a179-197abca7b738 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:11.972066 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 208/421] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:34:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:11.972066 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e01e231-72a3-43d4-a179-197abca7b738 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:12.028324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:12.038224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f08016f-b1d6-4b55-9ab1-d5c2925691d4 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:12.088742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0bb2fe-87f2-45f6-b102-b0f65b2b2453 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:12.089387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0bb2fe-87f2-45f6-b102-b0f65b2b2453 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:12.089678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0bb2fe-87f2-45f6-b102-b0f65b2b2453 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:12.089960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0bb2fe-87f2-45f6-b102-b0f65b2b2453 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:12.090711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0bb2fe-87f2-45f6-b102-b0f65b2b2453 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:12.091013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0bb2fe-87f2-45f6-b102-b0f65b2b2453 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:12.092416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0bb2fe-87f2-45f6-b102-b0f65b2b2453 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:12.102610 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 214/422] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:12.103528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0bb2fe-87f2-45f6-b102-b0f65b2b2453 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:12.109202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e01e231-72a3-43d4-a179-197abca7b738 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:12.162560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:12.162985 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74373b41-ff18-4227-a822-b28ddbeee47f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:12.163884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74373b41-ff18-4227-a822-b28ddbeee47f None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:12.164359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74373b41-ff18-4227-a822-b28ddbeee47f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:12.164855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74373b41-ff18-4227-a822-b28ddbeee47f None admin] PATH_INFO: `/v3/projects/25a26196f3ac436c876fd26eb5482e4a/groups/316757d7a0b34e3d94f7c18a30d1a3fa/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:12.169781 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74373b41-ff18-4227-a822-b28ddbeee47f None admin] RBAC: Authorizing `identity:create_grant(project_id=25a26196f3ac436c876fd26eb5482e4a, group_id=316757d7a0b34e3d94f7c18a30d1a3fa, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:12.172166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74373b41-ff18-4227-a822-b28ddbeee47f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:12.196635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 209/423] 10.222.0.40 () {58 vars in 1442 bytes} [Fri Jul 29 04:34:11 2022] PUT /identity/v3/projects/25a26196f3ac436c876fd26eb5482e4a/groups/316757d7a0b34e3d94f7c18a30d1a3fa/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:12.197772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74373b41-ff18-4227-a822-b28ddbeee47f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:12.251748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0bb2fe-87f2-45f6-b102-b0f65b2b2453 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:12.320463 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08564be-064f-49f7-95cb-1d4f2d40f124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:12.321826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08564be-064f-49f7-95cb-1d4f2d40f124 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:12.322115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08564be-064f-49f7-95cb-1d4f2d40f124 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:12.322394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08564be-064f-49f7-95cb-1d4f2d40f124 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:12.391819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74373b41-ff18-4227-a822-b28ddbeee47f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:12.395667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:12.443072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08564be-064f-49f7-95cb-1d4f2d40f124 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:12.444874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08564be-064f-49f7-95cb-1d4f2d40f124 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:12.445152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c96ac9-5f5b-44ff-805d-aab497fa0a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:12.445888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c96ac9-5f5b-44ff-805d-aab497fa0a1b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:12.446278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c96ac9-5f5b-44ff-805d-aab497fa0a1b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:12.446661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77c96ac9-5f5b-44ff-805d-aab497fa0a1b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:12.517226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:12.533496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c96ac9-5f5b-44ff-805d-aab497fa0a1b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:12.535294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77c96ac9-5f5b-44ff-805d-aab497fa0a1b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:12.583657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 215/424] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:12.590424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08564be-064f-49f7-95cb-1d4f2d40f124 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:12.764951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 210/425] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 573 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:12.766365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08564be-064f-49f7-95cb-1d4f2d40f124 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:12.770824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c96ac9-5f5b-44ff-805d-aab497fa0a1b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:12.777167 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:12.831446 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9247f26f-56c7-4589-a368-f2f6c9bb44e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:12.832448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9247f26f-56c7-4589-a368-f2f6c9bb44e6 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:12.833803 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9247f26f-56c7-4589-a368-f2f6c9bb44e6 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:12.834228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9247f26f-56c7-4589-a368-f2f6c9bb44e6 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:12.836849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9247f26f-56c7-4589-a368-f2f6c9bb44e6 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:12.838467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9247f26f-56c7-4589-a368-f2f6c9bb44e6 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:12.844956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 216/426] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:12 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:12.847175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9247f26f-56c7-4589-a368-f2f6c9bb44e6 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:12.868877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:12.926076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77c96ac9-5f5b-44ff-805d-aab497fa0a1b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:12.971455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7843fd19-d95d-4dca-8943-fe48f902df90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:12.973334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7843fd19-d95d-4dca-8943-fe48f902df90 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:12.973982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7843fd19-d95d-4dca-8943-fe48f902df90 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:12.975705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7843fd19-d95d-4dca-8943-fe48f902df90 None admin] PATH_INFO: `/v3/projects/e96107c2d25740e783b49bd3a96cbc8d` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:12.977274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7843fd19-d95d-4dca-8943-fe48f902df90 None admin] RBAC: Authorizing `identity:get_project(project_id=e96107c2d25740e783b49bd3a96cbc8d)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:12.982047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7843fd19-d95d-4dca-8943-fe48f902df90 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:12.984081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 211/427] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:12 2022] GET /identity/v3/projects/e96107c2d25740e783b49bd3a96cbc8d => generated 298 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:12.989952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7843fd19-d95d-4dca-8943-fe48f902df90 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:13.005528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9247f26f-56c7-4589-a368-f2f6c9bb44e6 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:13.040711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf8ebde-1e2c-458c-82da-3648b688a5f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:13.041366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf8ebde-1e2c-458c-82da-3648b688a5f5 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:13.041632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf8ebde-1e2c-458c-82da-3648b688a5f5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:13.041910 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf8ebde-1e2c-458c-82da-3648b688a5f5 None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:13.043146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf8ebde-1e2c-458c-82da-3648b688a5f5 None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:13.044659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf8ebde-1e2c-458c-82da-3648b688a5f5 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:13.046612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 217/428] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:12 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:13.053236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf8ebde-1e2c-458c-82da-3648b688a5f5 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:13.121124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:13.153095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:13.157382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7843fd19-d95d-4dca-8943-fe48f902df90 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:13.172113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf8ebde-1e2c-458c-82da-3648b688a5f5 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:13.196223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebd7527-2761-4478-9e50-a429f2a5f975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:13.197053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd7527-2761-4478-9e50-a429f2a5f975 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:13.197440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd7527-2761-4478-9e50-a429f2a5f975 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:13.197823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd7527-2761-4478-9e50-a429f2a5f975 None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:13.199095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebd7527-2761-4478-9e50-a429f2a5f975 None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:13.200657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebd7527-2761-4478-9e50-a429f2a5f975 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:13.202670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 212/429] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:12 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:13.203997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae19ed2d-a7e5-4a67-8964-733313bc764c None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:13.204386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae19ed2d-a7e5-4a67-8964-733313bc764c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:13.204846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae19ed2d-a7e5-4a67-8964-733313bc764c None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:13.207176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 213/430] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:13.209111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae19ed2d-a7e5-4a67-8964-733313bc764c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:13.224432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4cef4f-a69d-4340-a5e2-5d0e5080b46d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:13.225257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4cef4f-a69d-4340-a5e2-5d0e5080b46d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:13.225614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4cef4f-a69d-4340-a5e2-5d0e5080b46d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:13.225961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4cef4f-a69d-4340-a5e2-5d0e5080b46d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:13.226854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4cef4f-a69d-4340-a5e2-5d0e5080b46d None admin] RBAC: Adding query filter params (group.id=9f588151b280456981cfc088281fb10b, role.id=9e77a799dd224c1480f51a54f3aed529, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:13.227238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4cef4f-a69d-4340-a5e2-5d0e5080b46d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:13.228830 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4cef4f-a69d-4340-a5e2-5d0e5080b46d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:13.235781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 218/431] 10.222.0.40 () {56 vars in 1376 bytes} [Fri Jul 29 04:34:13 2022] GET /identity/v3/role_assignments?group.id=9f588151b280456981cfc088281fb10b&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 611 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:13.238109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d1f682-74cb-4e09-80eb-0b82ac915b31 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:13.238477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d1f682-74cb-4e09-80eb-0b82ac915b31 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:13.238829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d1f682-74cb-4e09-80eb-0b82ac915b31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:13.259828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f6d1f682-74cb-4e09-80eb-0b82ac915b31 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:13.261246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f6d1f682-74cb-4e09-80eb-0b82ac915b31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:13.337631 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae19ed2d-a7e5-4a67-8964-733313bc764c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:13.391994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 219/432] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:13.408659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6e5bfb-acae-4fcc-ab0b-a8b2b11dac01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:13.409468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6e5bfb-acae-4fcc-ab0b-a8b2b11dac01 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:13.409757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6e5bfb-acae-4fcc-ab0b-a8b2b11dac01 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:13.410034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6e5bfb-acae-4fcc-ab0b-a8b2b11dac01 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:13.410921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6e5bfb-acae-4fcc-ab0b-a8b2b11dac01 None admin] RBAC: Adding query filter params (group.id=9f588151b280456981cfc088281fb10b, role.id=28ddf4e7cb7f4139a3b6a79077418921, scope.project.id=e96107c2d25740e783b49bd3a96cbc8d) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:13.411239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6e5bfb-acae-4fcc-ab0b-a8b2b11dac01 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:13.413994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d1f682-74cb-4e09-80eb-0b82ac915b31 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:13.416305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6e5bfb-acae-4fcc-ab0b-a8b2b11dac01 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:13.424847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 214/433] 10.222.0.40 () {56 vars in 1376 bytes} [Fri Jul 29 04:34:13 2022] GET /identity/v3/role_assignments?group.id=9f588151b280456981cfc088281fb10b&scope.project.id=e96107c2d25740e783b49bd3a96cbc8d&role.id=28ddf4e7cb7f4139a3b6a79077418921 => generated 611 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:13.538696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:13.568459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d1f682-74cb-4e09-80eb-0b82ac915b31 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:13.620439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ad69af-f5c8-4bcd-882b-0eedea1fa759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:13.621138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ad69af-f5c8-4bcd-882b-0eedea1fa759 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:13.621426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ad69af-f5c8-4bcd-882b-0eedea1fa759 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:13.621703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ad69af-f5c8-4bcd-882b-0eedea1fa759 None admin] PATH_INFO: `/v3/roles/member` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:13.637937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ad69af-f5c8-4bcd-882b-0eedea1fa759 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-32ad69af-f5c8-4bcd-882b-0eedea1fa759 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:13.643207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:13.645303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:13.647252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 220/434] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:13.652929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6e5bfb-acae-4fcc-ab0b-a8b2b11dac01 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:13.658014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:13.698755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e11b98-ff27-45e5-8d5d-493f86ce3f85 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:13.698941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e11b98-ff27-45e5-8d5d-493f86ce3f85 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:13.699229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e11b98-ff27-45e5-8d5d-493f86ce3f85 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:13.700759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 221/435] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:34:13.706574 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2613dd6-ae6e-4a91-bdc1-29250bcc661a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:13.706920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2613dd6-ae6e-4a91-bdc1-29250bcc661a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:13.707201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2613dd6-ae6e-4a91-bdc1-29250bcc661a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:13.733722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c2613dd6-ae6e-4a91-bdc1-29250bcc661a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:13.735080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c2613dd6-ae6e-4a91-bdc1-29250bcc661a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:13.778453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6e5bfb-acae-4fcc-ab0b-a8b2b11dac01 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:13.818842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 222/436] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:13.829158 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8200f280-91f5-41d7-be81-bd874f70f0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:13.829788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8200f280-91f5-41d7-be81-bd874f70f0c3 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:13.830648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8200f280-91f5-41d7-be81-bd874f70f0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:13.831145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8200f280-91f5-41d7-be81-bd874f70f0c3 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:13.831879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8200f280-91f5-41d7-be81-bd874f70f0c3 None admin] RBAC: Adding query filter params (name=member) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:13.832191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8200f280-91f5-41d7-be81-bd874f70f0c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:13.833627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8200f280-91f5-41d7-be81-bd874f70f0c3 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:13.842528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 215/437] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:34:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:13.847945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2613dd6-ae6e-4a91-bdc1-29250bcc661a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:13.851676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8200f280-91f5-41d7-be81-bd874f70f0c3 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:13.940463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:13.942078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:13.980214 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2613dd6-ae6e-4a91-bdc1-29250bcc661a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:13.981536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8200f280-91f5-41d7-be81-bd874f70f0c3 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:14.028447 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9caf69cb-5bce-42ba-b2f3-f17ba5ca377f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:14.029239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caf69cb-5bce-42ba-b2f3-f17ba5ca377f None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.029420 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583f1106-f481-4667-9c27-b27db5690646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:14.029555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caf69cb-5bce-42ba-b2f3-f17ba5ca377f None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.029780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9caf69cb-5bce-42ba-b2f3-f17ba5ca377f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.030008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583f1106-f481-4667-9c27-b27db5690646 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.030296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583f1106-f481-4667-9c27-b27db5690646 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.030578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583f1106-f481-4667-9c27-b27db5690646 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.045378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9caf69cb-5bce-42ba-b2f3-f17ba5ca377f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-9caf69cb-5bce-42ba-b2f3-f17ba5ca377f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:14.050458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:14.051331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:14.052914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 223/438] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:34:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:14.057578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9caf69cb-5bce-42ba-b2f3-f17ba5ca377f None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:14.128823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583f1106-f481-4667-9c27-b27db5690646 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:14.131166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583f1106-f481-4667-9c27-b27db5690646 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:14.196705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9caf69cb-5bce-42ba-b2f3-f17ba5ca377f None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:14.219340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:14.230413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:14.238207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9dfa9d-49b4-4795-94e1-8b9854c7724d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:14.238843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9dfa9d-49b4-4795-94e1-8b9854c7724d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.239130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9dfa9d-49b4-4795-94e1-8b9854c7724d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.239407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9dfa9d-49b4-4795-94e1-8b9854c7724d None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.240129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9dfa9d-49b4-4795-94e1-8b9854c7724d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:14.240434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9dfa9d-49b4-4795-94e1-8b9854c7724d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:14.241835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9dfa9d-49b4-4795-94e1-8b9854c7724d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:14.250214 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 224/439] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:14 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:14.255538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9dfa9d-49b4-4795-94e1-8b9854c7724d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:14.260464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 216/440] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:14.265712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583f1106-f481-4667-9c27-b27db5690646 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:14.366089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9dfa9d-49b4-4795-94e1-8b9854c7724d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:14.368434 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583f1106-f481-4667-9c27-b27db5690646 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:14.401416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf3c7848-c651-4ea4-ae77-4ae93f46402a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:14.402082 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3c7848-c651-4ea4-ae77-4ae93f46402a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.402366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3c7848-c651-4ea4-ae77-4ae93f46402a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.402491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7fc4bd-19ed-4315-929f-07967e66fc04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:14.402689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf3c7848-c651-4ea4-ae77-4ae93f46402a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.403081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7fc4bd-19ed-4315-929f-07967e66fc04 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.403363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7fc4bd-19ed-4315-929f-07967e66fc04 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.403638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7fc4bd-19ed-4315-929f-07967e66fc04 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.404934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7fc4bd-19ed-4315-929f-07967e66fc04 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:14.406373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7fc4bd-19ed-4315-929f-07967e66fc04 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:14.408008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 217/441] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:14 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:14.414809 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7fc4bd-19ed-4315-929f-07967e66fc04 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:14.450604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:14.473006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:14.484486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3c7848-c651-4ea4-ae77-4ae93f46402a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:14.486699 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf3c7848-c651-4ea4-ae77-4ae93f46402a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:14.523207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7fc4bd-19ed-4315-929f-07967e66fc04 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:14.564047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ac1686-a199-4ee9-9a68-30a2bfe1bffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:14.565282 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ac1686-a199-4ee9-9a68-30a2bfe1bffa None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.565572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ac1686-a199-4ee9-9a68-30a2bfe1bffa None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.565861 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ac1686-a199-4ee9-9a68-30a2bfe1bffa None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.567059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ac1686-a199-4ee9-9a68-30a2bfe1bffa None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:14.568732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ac1686-a199-4ee9-9a68-30a2bfe1bffa None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:14.570347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 218/442] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:14 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:14.571565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6056e0ad-2212-4cac-bed9-57bf7b03a343 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.571912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6056e0ad-2212-4cac-bed9-57bf7b03a343 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.572200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6056e0ad-2212-4cac-bed9-57bf7b03a343 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.573546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 219/443] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:14.576708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6056e0ad-2212-4cac-bed9-57bf7b03a343 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:14.599430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 225/444] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 29 04:34:14.600798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e78cbf-9493-4361-937d-4bc4cfe58b36 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.601120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e78cbf-9493-4361-937d-4bc4cfe58b36 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.601402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e78cbf-9493-4361-937d-4bc4cfe58b36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.619323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-58e78cbf-9493-4361-937d-4bc4cfe58b36 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:14.620516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-58e78cbf-9493-4361-937d-4bc4cfe58b36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:14.683396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6056e0ad-2212-4cac-bed9-57bf7b03a343 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:14.684944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:14.690634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 226/445] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:14.691474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e78cbf-9493-4361-937d-4bc4cfe58b36 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:14.715963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:14.721273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc76021-a575-4619-9ea3-3e71c7473555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:14.721898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc76021-a575-4619-9ea3-3e71c7473555 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.722182 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc76021-a575-4619-9ea3-3e71c7473555 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.722460 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc76021-a575-4619-9ea3-3e71c7473555 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.723243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc76021-a575-4619-9ea3-3e71c7473555 None admin] RBAC: Adding query filter params (group.id=9f588151b280456981cfc088281fb10b, role.id=9e77a799dd224c1480f51a54f3aed529, scope.project.id=7a2a97f3a55842ac8262f51748e570c2) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:14.723558 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc76021-a575-4619-9ea3-3e71c7473555 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:14.725030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc76021-a575-4619-9ea3-3e71c7473555 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:14.730800 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 220/446] 10.222.0.40 () {56 vars in 1376 bytes} [Fri Jul 29 04:34:14 2022] GET /identity/v3/role_assignments?group.id=9f588151b280456981cfc088281fb10b&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=9e77a799dd224c1480f51a54f3aed529 => generated 611 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:14.731591 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc76021-a575-4619-9ea3-3e71c7473555 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:14.811910 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e78cbf-9493-4361-937d-4bc4cfe58b36 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:14.852843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb9b26e-cc93-4969-ac20-a65c5b0ff997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:14.853510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb9b26e-cc93-4969-ac20-a65c5b0ff997 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.853796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb9b26e-cc93-4969-ac20-a65c5b0ff997 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.854075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb9b26e-cc93-4969-ac20-a65c5b0ff997 None admin] PATH_INFO: `/v3/projects/7a2a97f3a55842ac8262f51748e570c2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.854194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc76021-a575-4619-9ea3-3e71c7473555 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:14.855350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb9b26e-cc93-4969-ac20-a65c5b0ff997 None admin] RBAC: Authorizing `identity:get_project(project_id=7a2a97f3a55842ac8262f51748e570c2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:14.856852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb9b26e-cc93-4969-ac20-a65c5b0ff997 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:14.858702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 227/447] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:14 2022] GET /identity/v3/projects/7a2a97f3a55842ac8262f51748e570c2 => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:14.864569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb9b26e-cc93-4969-ac20-a65c5b0ff997 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:14.900299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f943d78c-a863-4483-afb5-37b8fd2903f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:14.901094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f943d78c-a863-4483-afb5-37b8fd2903f5 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:14.901506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f943d78c-a863-4483-afb5-37b8fd2903f5 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:14.901791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f943d78c-a863-4483-afb5-37b8fd2903f5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:14.915248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f943d78c-a863-4483-afb5-37b8fd2903f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-f943d78c-a863-4483-afb5-37b8fd2903f5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:14.920725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:14.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:14.922731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 221/448] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:14.928702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f943d78c-a863-4483-afb5-37b8fd2903f5 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:14.963371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:14.972289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:14.976055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb9b26e-cc93-4969-ac20-a65c5b0ff997 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:15.012031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836acc61-f6f7-4af6-9d09-d43545b6f4e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:15.012709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836acc61-f6f7-4af6-9d09-d43545b6f4e8 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:15.012995 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836acc61-f6f7-4af6-9d09-d43545b6f4e8 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:15.013273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836acc61-f6f7-4af6-9d09-d43545b6f4e8 None admin] PATH_INFO: `/v3/groups/9f588151b280456981cfc088281fb10b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:15.014432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836acc61-f6f7-4af6-9d09-d43545b6f4e8 None admin] RBAC: Authorizing `identity:get_group(group_id=9f588151b280456981cfc088281fb10b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:15.016273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836acc61-f6f7-4af6-9d09-d43545b6f4e8 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:15.018374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 228/449] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:14 2022] GET /identity/v3/groups/9f588151b280456981cfc088281fb10b => generated 226 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:15.023960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836acc61-f6f7-4af6-9d09-d43545b6f4e8 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:15.062277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f943d78c-a863-4483-afb5-37b8fd2903f5 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:15.101223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceaa333-d7ca-429d-845b-c1e353e3ee40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:15.101846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceaa333-d7ca-429d-845b-c1e353e3ee40 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:15.102130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceaa333-d7ca-429d-845b-c1e353e3ee40 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:15.102407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceaa333-d7ca-429d-845b-c1e353e3ee40 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:15.103157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceaa333-d7ca-429d-845b-c1e353e3ee40 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:15.103467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceaa333-d7ca-429d-845b-c1e353e3ee40 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:15.104925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aceaa333-d7ca-429d-845b-c1e353e3ee40 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:15.113519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 222/450] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:15.118931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceaa333-d7ca-429d-845b-c1e353e3ee40 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:15.134752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836acc61-f6f7-4af6-9d09-d43545b6f4e8 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:15.168887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9688d65c-2172-42d6-9759-0e93497a7e2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:15.169498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9688d65c-2172-42d6-9759-0e93497a7e2c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:15.169781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9688d65c-2172-42d6-9759-0e93497a7e2c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:15.170062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9688d65c-2172-42d6-9759-0e93497a7e2c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:15.170850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9688d65c-2172-42d6-9759-0e93497a7e2c None admin] RBAC: Adding query filter params (group.id=9f588151b280456981cfc088281fb10b, role.id=28ddf4e7cb7f4139a3b6a79077418921, scope.project.id=7a2a97f3a55842ac8262f51748e570c2) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:15.171164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9688d65c-2172-42d6-9759-0e93497a7e2c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:15.172627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9688d65c-2172-42d6-9759-0e93497a7e2c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:15.178647 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 229/451] 10.222.0.40 () {56 vars in 1376 bytes} [Fri Jul 29 04:34:15 2022] GET /identity/v3/role_assignments?group.id=9f588151b280456981cfc088281fb10b&scope.project.id=7a2a97f3a55842ac8262f51748e570c2&role.id=28ddf4e7cb7f4139a3b6a79077418921 => generated 611 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:15.210676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:15.223685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aceaa333-d7ca-429d-845b-c1e353e3ee40 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:15.260081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a3a752-d157-4ea4-aa35-4ec1ea4a6c38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:15.260751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3a752-d157-4ea4-aa35-4ec1ea4a6c38 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:15.261041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3a752-d157-4ea4-aa35-4ec1ea4a6c38 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:15.261321 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3a752-d157-4ea4-aa35-4ec1ea4a6c38 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:15.333609 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a3a752-d157-4ea4-aa35-4ec1ea4a6c38 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:15.335785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a3a752-d157-4ea4-aa35-4ec1ea4a6c38 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:15.425103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:15.438127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 223/452] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 29 04:34:15.443994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9688d65c-2172-42d6-9759-0e93497a7e2c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:15.474688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:15.551968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9688d65c-2172-42d6-9759-0e93497a7e2c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:15.584194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f154efde-b84c-427f-bb25-02935f5fb38c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:15.584855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f154efde-b84c-427f-bb25-02935f5fb38c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:15.585140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f154efde-b84c-427f-bb25-02935f5fb38c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:15.585418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f154efde-b84c-427f-bb25-02935f5fb38c None admin] PATH_INFO: `/v3/projects/25a26196f3ac436c876fd26eb5482e4a` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:15.586559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f154efde-b84c-427f-bb25-02935f5fb38c None admin] RBAC: Authorizing `identity:get_project(project_id=25a26196f3ac436c876fd26eb5482e4a)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:15.587999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f154efde-b84c-427f-bb25-02935f5fb38c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:15.589658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 230/453] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:34:15 2022] GET /identity/v3/projects/25a26196f3ac436c876fd26eb5482e4a => generated 344 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:15.595248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a3a752-d157-4ea4-aa35-4ec1ea4a6c38 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:15.692168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a3a752-d157-4ea4-aa35-4ec1ea4a6c38 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:15.725311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d4d529-72b9-43cd-9b95-1bab61a0316c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:15.725881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4d529-72b9-43cd-9b95-1bab61a0316c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:15.726165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4d529-72b9-43cd-9b95-1bab61a0316c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:15.726443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4d529-72b9-43cd-9b95-1bab61a0316c None admin] PATH_INFO: `/v3/groups/316757d7a0b34e3d94f7c18a30d1a3fa` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:15.727572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d4d529-72b9-43cd-9b95-1bab61a0316c None admin] RBAC: Authorizing `identity:get_group(group_id=316757d7a0b34e3d94f7c18a30d1a3fa)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:15.729029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d4d529-72b9-43cd-9b95-1bab61a0316c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:15.730598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 224/454] 10.222.0.40 () {56 vars in 1180 bytes} [Fri Jul 29 04:34:15 2022] GET /identity/v3/groups/316757d7a0b34e3d94f7c18a30d1a3fa => generated 229 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:15.737084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f154efde-b84c-427f-bb25-02935f5fb38c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:15.823510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:15.833344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f154efde-b84c-427f-bb25-02935f5fb38c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:15.865601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6476dd53-98d1-46f7-b0fe-667c948affa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:15.866200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6476dd53-98d1-46f7-b0fe-667c948affa8 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:15.866482 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6476dd53-98d1-46f7-b0fe-667c948affa8 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:15.866757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6476dd53-98d1-46f7-b0fe-667c948affa8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:15.867539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6476dd53-98d1-46f7-b0fe-667c948affa8 None admin] RBAC: Adding query filter params (group.id=316757d7a0b34e3d94f7c18a30d1a3fa, role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=25a26196f3ac436c876fd26eb5482e4a) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:15.867854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6476dd53-98d1-46f7-b0fe-667c948affa8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:15.869305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6476dd53-98d1-46f7-b0fe-667c948affa8 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:15.874840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 231/455] 10.222.0.40 () {56 vars in 1376 bytes} [Fri Jul 29 04:34:15 2022] GET /identity/v3/role_assignments?group.id=316757d7a0b34e3d94f7c18a30d1a3fa&scope.project.id=25a26196f3ac436c876fd26eb5482e4a&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 611 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:18.418017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ccf0c-f939-4c6a-986a-668e1c8df0af None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:18.418893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ccf0c-f939-4c6a-986a-668e1c8df0af None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:18.419323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1ccf0c-f939-4c6a-986a-668e1c8df0af None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:18.421077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 225/456] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:18.427491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f398cf9e-603d-4d3a-a168-ac5471ee3721 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:18.428349 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f398cf9e-603d-4d3a-a168-ac5471ee3721 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:18.428867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f398cf9e-603d-4d3a-a168-ac5471ee3721 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:18.452462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f398cf9e-603d-4d3a-a168-ac5471ee3721 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:18.453710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f398cf9e-603d-4d3a-a168-ac5471ee3721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:18.472053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a47f05-221c-4ec4-bf16-695032b41109 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:18.472739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a47f05-221c-4ec4-bf16-695032b41109 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:18.473263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a47f05-221c-4ec4-bf16-695032b41109 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:18.475075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 226/457] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:18.479888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d66c94-78cb-4de6-a162-c274cfddd688 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:18.480352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d66c94-78cb-4de6-a162-c274cfddd688 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:18.480825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d66c94-78cb-4de6-a162-c274cfddd688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:18.512701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-b8d66c94-78cb-4de6-a162-c274cfddd688 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:18.514127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b8d66c94-78cb-4de6-a162-c274cfddd688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:18.539453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 232/458] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:18.540929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226b80ab-6ce4-49ff-8973-2541be1979cb None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:18.541223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226b80ab-6ce4-49ff-8973-2541be1979cb None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:18.541502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226b80ab-6ce4-49ff-8973-2541be1979cb None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:18.543007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 233/459] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:18.549826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f3b41c-3d46-4bee-9616-f6dd8c464ec3 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:18.550109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f3b41c-3d46-4bee-9616-f6dd8c464ec3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:18.550388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f3b41c-3d46-4bee-9616-f6dd8c464ec3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:18.573352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-b7f3b41c-3d46-4bee-9616-f6dd8c464ec3 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:18.574750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b7f3b41c-3d46-4bee-9616-f6dd8c464ec3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:18.583991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:18.607699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 227/460] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:18.609014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d66c94-78cb-4de6-a162-c274cfddd688 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:18.674315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 234/461] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:18.675477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f3b41c-3d46-4bee-9616-f6dd8c464ec3 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:18.693256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:18.763858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d66c94-78cb-4de6-a162-c274cfddd688 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:18.790997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f3b41c-3d46-4bee-9616-f6dd8c464ec3 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:18.809685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec760580-eb7a-49af-835a-f8c6413438a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:18.810342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec760580-eb7a-49af-835a-f8c6413438a9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:18.810630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec760580-eb7a-49af-835a-f8c6413438a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:18.810960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec760580-eb7a-49af-835a-f8c6413438a9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:18.824936 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec760580-eb7a-49af-835a-f8c6413438a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:18.827211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec760580-eb7a-49af-835a-f8c6413438a9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:18.830732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8478619e-9aad-4aa0-9e4d-c4e718124780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:18.831611 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8478619e-9aad-4aa0-9e4d-c4e718124780 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:18.832088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8478619e-9aad-4aa0-9e4d-c4e718124780 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:18.832557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8478619e-9aad-4aa0-9e4d-c4e718124780 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:18.843375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8478619e-9aad-4aa0-9e4d-c4e718124780 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:18.845133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8478619e-9aad-4aa0-9e4d-c4e718124780 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-ec760580-eb7a-49af-835a-f8c6413438a9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:18.846649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:18.847472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:18.848286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:18.848286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:18.848286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:18.848286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:18.848286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 228/462] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:18.849150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec760580-eb7a-49af-835a-f8c6413438a9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-8478619e-9aad-4aa0-9e4d-c4e718124780 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:18.864757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:18.865660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:18.866497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:18.866497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:18.866497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:18.866497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:18.873393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 235/463] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:18.874528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8478619e-9aad-4aa0-9e4d-c4e718124780 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:18.900916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:18.953631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:18.967527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec760580-eb7a-49af-835a-f8c6413438a9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:18.994305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8478619e-9aad-4aa0-9e4d-c4e718124780 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:19.004480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2be4429-51a5-4a60-a209-a6721f345e0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:19.005498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2be4429-51a5-4a60-a209-a6721f345e0b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.005999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2be4429-51a5-4a60-a209-a6721f345e0b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.006468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2be4429-51a5-4a60-a209-a6721f345e0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.018035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2be4429-51a5-4a60-a209-a6721f345e0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:19.019935 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2be4429-51a5-4a60-a209-a6721f345e0b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:19.032306 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9476ea8-c45e-4935-8660-91f79d40456f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:19.032989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9476ea8-c45e-4935-8660-91f79d40456f None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.033280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9476ea8-c45e-4935-8660-91f79d40456f None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.033570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9476ea8-c45e-4935-8660-91f79d40456f None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.034273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9476ea8-c45e-4935-8660-91f79d40456f None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:19.034584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9476ea8-c45e-4935-8660-91f79d40456f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:19.035960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9476ea8-c45e-4935-8660-91f79d40456f None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-e2be4429-51a5-4a60-a209-a6721f345e0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:19.037429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:19.038222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:19.039078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:19.039078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:19.039078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:19.039078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:19.041006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 229/464] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:19.041957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2be4429-51a5-4a60-a209-a6721f345e0b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:19.053144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 236/465] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:19.054225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9476ea8-c45e-4935-8660-91f79d40456f None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:19.161893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:19.169856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9476ea8-c45e-4935-8660-91f79d40456f None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:19.176880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2be4429-51a5-4a60-a209-a6721f345e0b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:19.201603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:19.212709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d02c75-c573-415c-a8aa-bb3be1b42ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:19.213272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d02c75-c573-415c-a8aa-bb3be1b42ece None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.213554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d02c75-c573-415c-a8aa-bb3be1b42ece None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.213827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d02c75-c573-415c-a8aa-bb3be1b42ece None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.214529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d02c75-c573-415c-a8aa-bb3be1b42ece None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:19.214625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58588a4d-efae-493e-a6f7-4281f4fa49a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:19.214836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d02c75-c573-415c-a8aa-bb3be1b42ece None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:19.215254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58588a4d-efae-493e-a6f7-4281f4fa49a7 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.215544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58588a4d-efae-493e-a6f7-4281f4fa49a7 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.215827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58588a4d-efae-493e-a6f7-4281f4fa49a7 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.216216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d02c75-c573-415c-a8aa-bb3be1b42ece None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:19.216529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58588a4d-efae-493e-a6f7-4281f4fa49a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:19.216865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58588a4d-efae-493e-a6f7-4281f4fa49a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:19.218237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58588a4d-efae-493e-a6f7-4281f4fa49a7 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:19.228226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 237/466] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:19.229036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d02c75-c573-415c-a8aa-bb3be1b42ece None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:19.245685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 230/467] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:19.245685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5329521f-2e97-4c7c-b8e4-04f83af50cce None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.245685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5329521f-2e97-4c7c-b8e4-04f83af50cce None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.245685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5329521f-2e97-4c7c-b8e4-04f83af50cce None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.245685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 231/468] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:19 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:19.245685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5329521f-2e97-4c7c-b8e4-04f83af50cce None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:19.346018 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d02c75-c573-415c-a8aa-bb3be1b42ece None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:19.359856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5329521f-2e97-4c7c-b8e4-04f83af50cce None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:19.384519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace7cf65-aece-4265-a1bf-c140aaba9312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:19.385182 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace7cf65-aece-4265-a1bf-c140aaba9312 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.385476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace7cf65-aece-4265-a1bf-c140aaba9312 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.385762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace7cf65-aece-4265-a1bf-c140aaba9312 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.386451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace7cf65-aece-4265-a1bf-c140aaba9312 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:19.387883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace7cf65-aece-4265-a1bf-c140aaba9312 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:19.395852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c94b6c-c8c5-458f-aaea-2f13a6197b50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:19.396470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c94b6c-c8c5-458f-aaea-2f13a6197b50 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.396775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c94b6c-c8c5-458f-aaea-2f13a6197b50 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.397394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c94b6c-c8c5-458f-aaea-2f13a6197b50 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.397720 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c94b6c-c8c5-458f-aaea-2f13a6197b50 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:19.399771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c94b6c-c8c5-458f-aaea-2f13a6197b50 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:19.410533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 238/469] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:34:19 2022] POST /identity/v3/users => generated 246 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:34:19.411177 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace7cf65-aece-4265-a1bf-c140aaba9312 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:19.445052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 232/470] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:34:19 2022] POST /identity/v3/users => generated 248 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:34:19.445052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49efe9-e276-4e56-99a9-e86551cd3c3f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.445052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49efe9-e276-4e56-99a9-e86551cd3c3f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.445052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da49efe9-e276-4e56-99a9-e86551cd3c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.472140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:19.483586 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-da49efe9-e276-4e56-99a9-e86551cd3c3f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:19.484906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-da49efe9-e276-4e56-99a9-e86551cd3c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:19.491505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:19.604010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace7cf65-aece-4265-a1bf-c140aaba9312 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:19.623656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 233/471] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:19.654646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da49efe9-e276-4e56-99a9-e86551cd3c3f None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:19.659558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d198b130-626e-4247-9488-9ba589dab722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:19.659558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d198b130-626e-4247-9488-9ba589dab722 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.659558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d198b130-626e-4247-9488-9ba589dab722 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.659558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d198b130-626e-4247-9488-9ba589dab722 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.659889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d198b130-626e-4247-9488-9ba589dab722 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:19.661069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d198b130-626e-4247-9488-9ba589dab722 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:19.692176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 239/472] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:34:19 2022] POST /identity/v3/users => generated 248 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:34:19.760488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b4f616-e9e5-475e-8554-35f2e7cf26c2 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.760876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b4f616-e9e5-475e-8554-35f2e7cf26c2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.761100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b4f616-e9e5-475e-8554-35f2e7cf26c2 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.763945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 240/473] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:34:19.768757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40edf029-a7ad-4fc1-8797-6b767bec74b0 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.769047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40edf029-a7ad-4fc1-8797-6b767bec74b0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.769331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40edf029-a7ad-4fc1-8797-6b767bec74b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.789139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-40edf029-a7ad-4fc1-8797-6b767bec74b0 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:19.789490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da49efe9-e276-4e56-99a9-e86551cd3c3f None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:19.790316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-40edf029-a7ad-4fc1-8797-6b767bec74b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:19.838386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fc40f9-ebec-414d-b51e-4749c51c74a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:19.839462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fc40f9-ebec-414d-b51e-4749c51c74a5 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:19.839990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fc40f9-ebec-414d-b51e-4749c51c74a5 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:19.840274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fc40f9-ebec-414d-b51e-4749c51c74a5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:19.843931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:19.856006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fc40f9-ebec-414d-b51e-4749c51c74a5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:19.857546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fc40f9-ebec-414d-b51e-4749c51c74a5 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c6fc40f9-ebec-414d-b51e-4749c51c74a5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:19.877421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:19.878870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:19.879953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:19.879953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:19.879953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:19.879953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:19.879953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 234/474] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:19.883857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fc40f9-ebec-414d-b51e-4749c51c74a5 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:19.884419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 241/475] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:34:19.906296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40edf029-a7ad-4fc1-8797-6b767bec74b0 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:19.943883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:20.003494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fc40f9-ebec-414d-b51e-4749c51c74a5 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:20.035849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40edf029-a7ad-4fc1-8797-6b767bec74b0 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:20.047450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaf9a12-74c7-4a77-92d5-07f070cf2457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:20.048061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaf9a12-74c7-4a77-92d5-07f070cf2457 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:20.048924 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaf9a12-74c7-4a77-92d5-07f070cf2457 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:20.049102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbaf9a12-74c7-4a77-92d5-07f070cf2457 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:20.049914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaf9a12-74c7-4a77-92d5-07f070cf2457 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:20.050225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaf9a12-74c7-4a77-92d5-07f070cf2457 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:20.051767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbaf9a12-74c7-4a77-92d5-07f070cf2457 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:20.065908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 235/476] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:20.072527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaf9a12-74c7-4a77-92d5-07f070cf2457 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:20.076659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c245711e-ccb8-4f7c-bbcb-908f64df1b86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:20.077303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c245711e-ccb8-4f7c-bbcb-908f64df1b86 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:20.077590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c245711e-ccb8-4f7c-bbcb-908f64df1b86 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:20.077867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c245711e-ccb8-4f7c-bbcb-908f64df1b86 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:20.078598 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c245711e-ccb8-4f7c-bbcb-908f64df1b86 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:20.080695 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c245711e-ccb8-4f7c-bbcb-908f64df1b86 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:20.090986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.sql.core [None req-c245711e-ccb8-4f7c-bbcb-908f64df1b86 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 29 04:34:20.090986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 29 04:34:20.090986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [parameters: {'id': '1613d47891704f96b6a3410c9bd46153', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 29 04:34:20.090986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=92190) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-c245711e-ccb8-4f7c-bbcb-908f64df1b86 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application result = self._query(query) Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application conn.query(q) Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application result.read() Jul 29 04:34:20.115589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return ref.to_dict() Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 29 04:34:20.116484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application next(self.gen) Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application yield resource Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application next(self.gen) Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self.session.rollback() Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self.force_reraise() Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise self.value Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application session.commit() Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 29 04:34:20.117401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._prepare_impl() Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self.session.flush() Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._flush(objects) Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application compat.raise_( Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application flush_context.execute() Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rec.execute(self) Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 29 04:34:20.118268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 29 04:34:20.119118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application result = self._query(query) Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application conn.query(q) Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application result.read() Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 29 04:34:20.119989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [parameters: {'id': '1613d47891704f96b6a3410c9bd46153', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:20.122047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 29 04:34:20.124240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:20.133256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 242/477] 10.222.0.40 () {60 vars in 1131 bytes} [Fri Jul 29 04:34:19 2022] POST /identity/v3/roles => generated 144 bytes in 212 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:20.133256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c245711e-ccb8-4f7c-bbcb-908f64df1b86 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:20.134746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:20.224707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbaf9a12-74c7-4a77-92d5-07f070cf2457 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:20.268460 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:20.280418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c245711e-ccb8-4f7c-bbcb-908f64df1b86 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:20.309401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f5953f-7415-4cf9-8ec0-e538c5312663 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:20.310394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f5953f-7415-4cf9-8ec0-e538c5312663 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:20.310874 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f5953f-7415-4cf9-8ec0-e538c5312663 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:20.311352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f5953f-7415-4cf9-8ec0-e538c5312663 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:20.312227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f5953f-7415-4cf9-8ec0-e538c5312663 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:20.313907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f5953f-7415-4cf9-8ec0-e538c5312663 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:20.322162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e13e310-1b8b-4c7f-b002-9e0909bd6a6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:20.322799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e13e310-1b8b-4c7f-b002-9e0909bd6a6e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:20.323086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e13e310-1b8b-4c7f-b002-9e0909bd6a6e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:20.323367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e13e310-1b8b-4c7f-b002-9e0909bd6a6e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:20.336152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e13e310-1b8b-4c7f-b002-9e0909bd6a6e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-9e13e310-1b8b-4c7f-b002-9e0909bd6a6e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:20.341874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:20.342783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:20.343556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 243/478] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:34:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:20.353540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e13e310-1b8b-4c7f-b002-9e0909bd6a6e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:20.360223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 236/479] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:34:20 2022] POST /identity/v3/users => generated 249 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:34:20.481816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e13e310-1b8b-4c7f-b002-9e0909bd6a6e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:20.528631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5b72ef-985f-40af-955e-feb3ec7d12f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:20.529289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5b72ef-985f-40af-955e-feb3ec7d12f0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:20.529607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5b72ef-985f-40af-955e-feb3ec7d12f0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:20.529896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5b72ef-985f-40af-955e-feb3ec7d12f0 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:20.530640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5b72ef-985f-40af-955e-feb3ec7d12f0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:20.530955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5b72ef-985f-40af-955e-feb3ec7d12f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:20.532355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5b72ef-985f-40af-955e-feb3ec7d12f0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:20.543606 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 244/480] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:22.015599 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15b71f6-5fe1-4dde-8d3a-e1ae6650b088 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.016642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15b71f6-5fe1-4dde-8d3a-e1ae6650b088 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.017160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15b71f6-5fe1-4dde-8d3a-e1ae6650b088 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.019022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 237/481] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:22.024434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6a399c-51de-4610-8314-982c332eb157 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.026381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96037ed-f35a-4c81-8083-6098595abf45 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.026863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96037ed-f35a-4c81-8083-6098595abf45 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.027332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96037ed-f35a-4c81-8083-6098595abf45 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.029079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 238/482] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:22.029590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6a399c-51de-4610-8314-982c332eb157 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.030062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6a399c-51de-4610-8314-982c332eb157 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.034609 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb510abe-7896-4f72-a494-8f6ea50e6a72 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.035169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb510abe-7896-4f72-a494-8f6ea50e6a72 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.035648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb510abe-7896-4f72-a494-8f6ea50e6a72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.069323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-bb510abe-7896-4f72-a494-8f6ea50e6a72 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:22.070583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-8b6a399c-51de-4610-8314-982c332eb157 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:22.072041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-bb510abe-7896-4f72-a494-8f6ea50e6a72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:22.073422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8b6a399c-51de-4610-8314-982c332eb157 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:22.092506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:22.191723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 245/483] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:22.201299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 239/484] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:34:22.215766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6a399c-51de-4610-8314-982c332eb157 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:22.245665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb510abe-7896-4f72-a494-8f6ea50e6a72 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:22.297716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:22.382606 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6a399c-51de-4610-8314-982c332eb157 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:22.415833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb510abe-7896-4f72-a494-8f6ea50e6a72 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:22.443701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249a5948-f927-4c42-93e3-6cb7b97fca4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:22.444342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a5948-f927-4c42-93e3-6cb7b97fca4d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.444632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a5948-f927-4c42-93e3-6cb7b97fca4d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.444912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249a5948-f927-4c42-93e3-6cb7b97fca4d None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.464216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249a5948-f927-4c42-93e3-6cb7b97fca4d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-249a5948-f927-4c42-93e3-6cb7b97fca4d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:22.469079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:22.469937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:22.471370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:22.471959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 246/485] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:22.476358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba10bab3-6817-411a-adc0-74207d52d12b None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.476768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba10bab3-6817-411a-adc0-74207d52d12b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.477059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba10bab3-6817-411a-adc0-74207d52d12b None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.478842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 247/486] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:34:22.479580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba10bab3-6817-411a-adc0-74207d52d12b None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:22.485040 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:22.488560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520edadc-95d2-40fe-a26f-dc70b0b34b3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:22.489259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520edadc-95d2-40fe-a26f-dc70b0b34b3a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.489548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520edadc-95d2-40fe-a26f-dc70b0b34b3a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.489828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520edadc-95d2-40fe-a26f-dc70b0b34b3a None admin] PATH_INFO: `/v3/roles/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.500801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520edadc-95d2-40fe-a26f-dc70b0b34b3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-520edadc-95d2-40fe-a26f-dc70b0b34b3a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:22.506471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:22.507329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:22.508095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:22.510767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 240/487] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:22.512190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecff604-c45e-481c-aa4a-1a3ce03ab566 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.512548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecff604-c45e-481c-aa4a-1a3ce03ab566 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.512911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecff604-c45e-481c-aa4a-1a3ce03ab566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.532277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0ecff604-c45e-481c-aa4a-1a3ce03ab566 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:22.534316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0ecff604-c45e-481c-aa4a-1a3ce03ab566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:22.606352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba10bab3-6817-411a-adc0-74207d52d12b None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:22.624651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 241/488] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:22.624651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecff604-c45e-481c-aa4a-1a3ce03ab566 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:22.647742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:22.673966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc41b405-916a-441d-842d-ac821571f810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:22.675046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc41b405-916a-441d-842d-ac821571f810 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.675513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc41b405-916a-441d-842d-ac821571f810 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.676030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc41b405-916a-441d-842d-ac821571f810 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.676976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc41b405-916a-441d-842d-ac821571f810 None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:22.677523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc41b405-916a-441d-842d-ac821571f810 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:22.679482 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc41b405-916a-441d-842d-ac821571f810 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:22.690295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 248/489] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:22.691421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc41b405-916a-441d-842d-ac821571f810 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:22.744536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecff604-c45e-481c-aa4a-1a3ce03ab566 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:22.779078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:22.784299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c01d3d-cbec-41c6-9ceb-b3d811de2a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:22.785973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c01d3d-cbec-41c6-9ceb-b3d811de2a94 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.786469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c01d3d-cbec-41c6-9ceb-b3d811de2a94 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.786969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c01d3d-cbec-41c6-9ceb-b3d811de2a94 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.787915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c01d3d-cbec-41c6-9ceb-b3d811de2a94 None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:22.788455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c01d3d-cbec-41c6-9ceb-b3d811de2a94 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:22.790073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c01d3d-cbec-41c6-9ceb-b3d811de2a94 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:22.800915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 242/490] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:22.802114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c01d3d-cbec-41c6-9ceb-b3d811de2a94 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:22.819266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc41b405-916a-441d-842d-ac821571f810 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:22.856769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491dfff5-3224-46e7-9ec4-4ededdadfe3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:22.857331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491dfff5-3224-46e7-9ec4-4ededdadfe3e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.857618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491dfff5-3224-46e7-9ec4-4ededdadfe3e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.857892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491dfff5-3224-46e7-9ec4-4ededdadfe3e None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.868917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491dfff5-3224-46e7-9ec4-4ededdadfe3e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-491dfff5-3224-46e7-9ec4-4ededdadfe3e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:22.874530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:22.875409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:22.876242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 249/491] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:22.881812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491dfff5-3224-46e7-9ec4-4ededdadfe3e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:22.914552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:22.934596 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c01d3d-cbec-41c6-9ceb-b3d811de2a94 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:22.976796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271f7043-5561-4bb9-baef-48b53dcee66b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:22.977450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271f7043-5561-4bb9-baef-48b53dcee66b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:22.977735 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271f7043-5561-4bb9-baef-48b53dcee66b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:22.978014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271f7043-5561-4bb9-baef-48b53dcee66b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:22.991196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271f7043-5561-4bb9-baef-48b53dcee66b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:22.992689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271f7043-5561-4bb9-baef-48b53dcee66b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-271f7043-5561-4bb9-baef-48b53dcee66b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:23.007424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:23.008295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:23.009166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:23.009166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:23.009166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:23.009166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:23.009166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 243/492] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:23.009455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271f7043-5561-4bb9-baef-48b53dcee66b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.016875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491dfff5-3224-46e7-9ec4-4ededdadfe3e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.045985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:23.056243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cafc34c-dbc4-496e-b36f-46da2fcc8562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.056916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cafc34c-dbc4-496e-b36f-46da2fcc8562 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.057202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cafc34c-dbc4-496e-b36f-46da2fcc8562 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.057484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cafc34c-dbc4-496e-b36f-46da2fcc8562 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.068251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cafc34c-dbc4-496e-b36f-46da2fcc8562 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:23.069727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cafc34c-dbc4-496e-b36f-46da2fcc8562 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-4cafc34c-dbc4-496e-b36f-46da2fcc8562 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:23.085100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:23.086101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:23.087119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:23.087119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:23.087119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:23.087119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:23.090937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 250/493] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:23.092204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39da8af9-a7f3-4a52-8d27-483da2378da3 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.092491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39da8af9-a7f3-4a52-8d27-483da2378da3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.092799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39da8af9-a7f3-4a52-8d27-483da2378da3 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.094198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 251/494] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:23.097377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39da8af9-a7f3-4a52-8d27-483da2378da3 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.124327 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271f7043-5561-4bb9-baef-48b53dcee66b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.160701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b9c05b-1f9c-419f-a4a7-28e0a8d6ad19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.161505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b9c05b-1f9c-419f-a4a7-28e0a8d6ad19 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.161794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b9c05b-1f9c-419f-a4a7-28e0a8d6ad19 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.162076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b9c05b-1f9c-419f-a4a7-28e0a8d6ad19 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.162795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b9c05b-1f9c-419f-a4a7-28e0a8d6ad19 None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:23.163103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b9c05b-1f9c-419f-a4a7-28e0a8d6ad19 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:23.164489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b9c05b-1f9c-419f-a4a7-28e0a8d6ad19 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.169553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:23.178239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 244/495] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:23.178239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b9c05b-1f9c-419f-a4a7-28e0a8d6ad19 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.212518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39da8af9-a7f3-4a52-8d27-483da2378da3 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.245819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd789e3-fc81-4d4d-9d37-64a43cd0d5a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.246526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd789e3-fc81-4d4d-9d37-64a43cd0d5a4 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.246806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd789e3-fc81-4d4d-9d37-64a43cd0d5a4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.247080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd789e3-fc81-4d4d-9d37-64a43cd0d5a4 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.247771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd789e3-fc81-4d4d-9d37-64a43cd0d5a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:23.248075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd789e3-fc81-4d4d-9d37-64a43cd0d5a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:23.249482 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd789e3-fc81-4d4d-9d37-64a43cd0d5a4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.261778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 252/496] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:23.262938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fb7592-61a8-4321-8529-766e213664a9 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.263222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fb7592-61a8-4321-8529-766e213664a9 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.263496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46fb7592-61a8-4321-8529-766e213664a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.280846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-46fb7592-61a8-4321-8529-766e213664a9 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:23.282033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-46fb7592-61a8-4321-8529-766e213664a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:23.282828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b9c05b-1f9c-419f-a4a7-28e0a8d6ad19 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.311110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:23.318095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ffabf4-2994-44e6-81b9-ff40065ac754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.318754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ffabf4-2994-44e6-81b9-ff40065ac754 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.319038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ffabf4-2994-44e6-81b9-ff40065ac754 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.319319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ffabf4-2994-44e6-81b9-ff40065ac754 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.320013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ffabf4-2994-44e6-81b9-ff40065ac754 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:23.320326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ffabf4-2994-44e6-81b9-ff40065ac754 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:23.321729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ffabf4-2994-44e6-81b9-ff40065ac754 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.334191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 245/497] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:23.335369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c3c244-70e2-47c0-99a5-b825a965cb86 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.335654 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c3c244-70e2-47c0-99a5-b825a965cb86 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.335965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c3c244-70e2-47c0-99a5-b825a965cb86 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.337360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 246/498] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:23 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:23.337874 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c3c244-70e2-47c0-99a5-b825a965cb86 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.351469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 253/499] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:23.352389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fb7592-61a8-4321-8529-766e213664a9 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.398402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:23.438679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c3c244-70e2-47c0-99a5-b825a965cb86 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.451249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46fb7592-61a8-4321-8529-766e213664a9 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.472743 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecae065-c274-46a9-afe9-fc24194e0542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.473499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecae065-c274-46a9-afe9-fc24194e0542 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.473884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecae065-c274-46a9-afe9-fc24194e0542 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.474267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecae065-c274-46a9-afe9-fc24194e0542 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.484054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feebea56-f9fd-4939-a866-71a87b1d8d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.484709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feebea56-f9fd-4939-a866-71a87b1d8d9d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.484807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecae065-c274-46a9-afe9-fc24194e0542 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:23.484993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feebea56-f9fd-4939-a866-71a87b1d8d9d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.485267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feebea56-f9fd-4939-a866-71a87b1d8d9d None admin] PATH_INFO: `/v3/users/nova` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.486210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecae065-c274-46a9-afe9-fc24194e0542 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.492282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feebea56-f9fd-4939-a866-71a87b1d8d9d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:23.493738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feebea56-f9fd-4939-a866-71a87b1d8d9d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-1ecae065-c274-46a9-afe9-fc24194e0542 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:23.500245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:23.501240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:23.502604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:23.502604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:23.502604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:23.502604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:23.502604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 247/500] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:23.503232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecae065-c274-46a9-afe9-fc24194e0542 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-feebea56-f9fd-4939-a866-71a87b1d8d9d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:23.506889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:23.507914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:23.508855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 254/501] 10.222.0.40 () {56 vars in 1093 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:23.509402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7ae50-b55b-4a93-ab04-3350890fe4ec None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.509464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7ae50-b55b-4a93-ab04-3350890fe4ec None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.509724 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c7ae50-b55b-4a93-ab04-3350890fe4ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.527454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c1c7ae50-b55b-4a93-ab04-3350890fe4ec None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:23.528769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c1c7ae50-b55b-4a93-ab04-3350890fe4ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:23.553372 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:23.596620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 255/502] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:23.597469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c7ae50-b55b-4a93-ab04-3350890fe4ec None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.617515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecae065-c274-46a9-afe9-fc24194e0542 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.631041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:23.653024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafacc1a-c1c6-4d0c-9595-f2ea15a93403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.653538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafacc1a-c1c6-4d0c-9595-f2ea15a93403 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.653817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafacc1a-c1c6-4d0c-9595-f2ea15a93403 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.654101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafacc1a-c1c6-4d0c-9595-f2ea15a93403 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.661633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafacc1a-c1c6-4d0c-9595-f2ea15a93403 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:23.663072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafacc1a-c1c6-4d0c-9595-f2ea15a93403 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-cafacc1a-c1c6-4d0c-9595-f2ea15a93403 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:23.676584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:23.677478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:23.678360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 248/503] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/users/cinder => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:23.678913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafacc1a-c1c6-4d0c-9595-f2ea15a93403 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.701964 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c7ae50-b55b-4a93-ab04-3350890fe4ec None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.735130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb906386-57e5-41ab-91d1-43eec54348ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.736287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb906386-57e5-41ab-91d1-43eec54348ec None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.736594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb906386-57e5-41ab-91d1-43eec54348ec None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.736983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb906386-57e5-41ab-91d1-43eec54348ec None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.746845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb906386-57e5-41ab-91d1-43eec54348ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-bb906386-57e5-41ab-91d1-43eec54348ec None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:23.752046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:23.752949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:23.753716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 256/504] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:23.754341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb906386-57e5-41ab-91d1-43eec54348ec None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.772712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:23.782888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafacc1a-c1c6-4d0c-9595-f2ea15a93403 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.816366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e302b0-e1fb-49d2-a4c1-d13b30a1b2e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.817100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e302b0-e1fb-49d2-a4c1-d13b30a1b2e0 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.817386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e302b0-e1fb-49d2-a4c1-d13b30a1b2e0 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.817661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e302b0-e1fb-49d2-a4c1-d13b30a1b2e0 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.818353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e302b0-e1fb-49d2-a4c1-d13b30a1b2e0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:23.818657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e302b0-e1fb-49d2-a4c1-d13b30a1b2e0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:23.820082 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e302b0-e1fb-49d2-a4c1-d13b30a1b2e0 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.832031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 249/505] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:23.832853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e302b0-e1fb-49d2-a4c1-d13b30a1b2e0 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.857504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb906386-57e5-41ab-91d1-43eec54348ec None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.864497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:23.891367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764f2e0e-3200-4d08-91ae-b9c0060feddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.891982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764f2e0e-3200-4d08-91ae-b9c0060feddc None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.892902 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764f2e0e-3200-4d08-91ae-b9c0060feddc None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.893162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764f2e0e-3200-4d08-91ae-b9c0060feddc None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.893947 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764f2e0e-3200-4d08-91ae-b9c0060feddc None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:23.894257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764f2e0e-3200-4d08-91ae-b9c0060feddc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:23.895691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764f2e0e-3200-4d08-91ae-b9c0060feddc None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.932680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 257/506] 10.222.0.40 () {56 vars in 1133 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:23.933032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e302b0-e1fb-49d2-a4c1-d13b30a1b2e0 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:23.933465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764f2e0e-3200-4d08-91ae-b9c0060feddc None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:23.967676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ef889a-45d5-4cee-9940-912cdf3837a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:23.968316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ef889a-45d5-4cee-9940-912cdf3837a8 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:23.968622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ef889a-45d5-4cee-9940-912cdf3837a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:23.968904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ef889a-45d5-4cee-9940-912cdf3837a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:23.979364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ef889a-45d5-4cee-9940-912cdf3837a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:23.980856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ef889a-45d5-4cee-9940-912cdf3837a8 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-94ef889a-45d5-4cee-9940-912cdf3837a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:23.996855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:23.997689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:23.998503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:23.998503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:23.998503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:23.998503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:23.998503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 250/507] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:23.998764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ef889a-45d5-4cee-9940-912cdf3837a8 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:24.016512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:24.039630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764f2e0e-3200-4d08-91ae-b9c0060feddc None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:24.072692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dacecf-653d-4d0d-89d4-2afad867deeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:24.073309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dacecf-653d-4d0d-89d4-2afad867deeb None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:24.073570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dacecf-653d-4d0d-89d4-2afad867deeb None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:24.073842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dacecf-653d-4d0d-89d4-2afad867deeb None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:24.074630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dacecf-653d-4d0d-89d4-2afad867deeb None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:24.074938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dacecf-653d-4d0d-89d4-2afad867deeb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:24.076368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dacecf-653d-4d0d-89d4-2afad867deeb None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:24.085272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:24.093288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 258/508] 10.222.0.40 () {56 vars in 1137 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:24.098563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dacecf-653d-4d0d-89d4-2afad867deeb None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:24.102444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ef889a-45d5-4cee-9940-912cdf3837a8 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:24.136616 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb2af9-4e11-403b-89dc-c4de59f4736f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:24.137250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb2af9-4e11-403b-89dc-c4de59f4736f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:24.137518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb2af9-4e11-403b-89dc-c4de59f4736f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:24.137798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb2af9-4e11-403b-89dc-c4de59f4736f None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:24.138520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb2af9-4e11-403b-89dc-c4de59f4736f None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:24.138830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb2af9-4e11-403b-89dc-c4de59f4736f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:24.140213 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb2af9-4e11-403b-89dc-c4de59f4736f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:24.149115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 251/509] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:24.150043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb2af9-4e11-403b-89dc-c4de59f4736f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:24.196153 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dacecf-653d-4d0d-89d4-2afad867deeb None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:24.229664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f53420-3711-4e06-a48a-df4f63d272e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:24.230319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f53420-3711-4e06-a48a-df4f63d272e3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:24.230603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f53420-3711-4e06-a48a-df4f63d272e3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:24.230883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f53420-3711-4e06-a48a-df4f63d272e3 None admin] PATH_INFO: `/v3/users/glance` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:24.232902 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:24.239201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f53420-3711-4e06-a48a-df4f63d272e3 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:24.240667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f53420-3711-4e06-a48a-df4f63d272e3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:24.250318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb2af9-4e11-403b-89dc-c4de59f4736f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-a7f53420-3711-4e06-a48a-df4f63d272e3 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:24.252861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:24.253765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:24.254782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 259/510] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:24 2022] GET /identity/v3/users/glance => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:24.255470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f53420-3711-4e06-a48a-df4f63d272e3 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:24.288456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7a8c8d-9c1e-4c8b-8172-94fa2863a245 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:24.289110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7a8c8d-9c1e-4c8b-8172-94fa2863a245 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:24.289395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7a8c8d-9c1e-4c8b-8172-94fa2863a245 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:24.289672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7a8c8d-9c1e-4c8b-8172-94fa2863a245 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:24.305410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:24.353585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f53420-3711-4e06-a48a-df4f63d272e3 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:24.357690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7a8c8d-9c1e-4c8b-8172-94fa2863a245 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:24.359144 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7a8c8d-9c1e-4c8b-8172-94fa2863a245 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:24.386616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6ec6b5-9860-478f-9a1a-349de7356313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:24.387225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6ec6b5-9860-478f-9a1a-349de7356313 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:24.387509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6ec6b5-9860-478f-9a1a-349de7356313 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:24.387786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6ec6b5-9860-478f-9a1a-349de7356313 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:24.388467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6ec6b5-9860-478f-9a1a-349de7356313 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:24.388811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6ec6b5-9860-478f-9a1a-349de7356313 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:24.390169 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6ec6b5-9860-478f-9a1a-349de7356313 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:24.401700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 260/511] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:24.402600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6ec6b5-9860-478f-9a1a-349de7356313 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:24.458533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:24.463731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 252/512] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:24.463731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7a8c8d-9c1e-4c8b-8172-94fa2863a245 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:24.504951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:24.507580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6ec6b5-9860-478f-9a1a-349de7356313 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:24.542173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4823af17-fff3-4fb4-932e-5d74f6b788c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:24.542815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4823af17-fff3-4fb4-932e-5d74f6b788c1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:24.543095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4823af17-fff3-4fb4-932e-5d74f6b788c1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:24.543365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4823af17-fff3-4fb4-932e-5d74f6b788c1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:24.561291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7a8c8d-9c1e-4c8b-8172-94fa2863a245 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:24.596493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47cc96-55ca-4632-8c05-3ef99b72323f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:24.597148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47cc96-55ca-4632-8c05-3ef99b72323f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:24.597456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47cc96-55ca-4632-8c05-3ef99b72323f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:24.597728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47cc96-55ca-4632-8c05-3ef99b72323f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:24.607670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47cc96-55ca-4632-8c05-3ef99b72323f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:24.609155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47cc96-55ca-4632-8c05-3ef99b72323f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:24.613996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4823af17-fff3-4fb4-932e-5d74f6b788c1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:24.615477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4823af17-fff3-4fb4-932e-5d74f6b788c1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-bb47cc96-55ca-4632-8c05-3ef99b72323f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:24.623835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:24.624845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:24.625731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:24.625731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:24.625731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:24.625731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:24.625731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 253/513] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:24.626245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47cc96-55ca-4632-8c05-3ef99b72323f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:24.676304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:24.721307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 261/514] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:24.722047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4823af17-fff3-4fb4-932e-5d74f6b788c1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:24.728221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:24.732165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47cc96-55ca-4632-8c05-3ef99b72323f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:24.765898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2231ea-1d25-48ee-83e1-46c97c327ddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:24.766515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2231ea-1d25-48ee-83e1-46c97c327ddf None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:24.766798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2231ea-1d25-48ee-83e1-46c97c327ddf None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:24.767667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2231ea-1d25-48ee-83e1-46c97c327ddf None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:24.768857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2231ea-1d25-48ee-83e1-46c97c327ddf None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:24.769193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2231ea-1d25-48ee-83e1-46c97c327ddf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:24.770647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2231ea-1d25-48ee-83e1-46c97c327ddf None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:24.805589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 254/515] 10.222.0.40 () {56 vars in 1137 bytes} [Fri Jul 29 04:34:24 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:24.806408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2231ea-1d25-48ee-83e1-46c97c327ddf None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:24.822733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4823af17-fff3-4fb4-932e-5d74f6b788c1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:24.856269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a3e4f6-673f-4338-86a2-571c330a15d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:24.856920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a3e4f6-673f-4338-86a2-571c330a15d1 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:24.857208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a3e4f6-673f-4338-86a2-571c330a15d1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:24.857486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a3e4f6-673f-4338-86a2-571c330a15d1 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:24.858160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a3e4f6-673f-4338-86a2-571c330a15d1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:24.859587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a3e4f6-673f-4338-86a2-571c330a15d1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:24.880648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 262/516] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:34:24 2022] POST /identity/v3/users => generated 247 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:34:24.881476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a3e4f6-673f-4338-86a2-571c330a15d1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:24.904839 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:24.916016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2231ea-1d25-48ee-83e1-46c97c327ddf None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:24.960981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a29f0c6-d1b9-45d8-869a-440ae1ccd9d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:24.961647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a29f0c6-d1b9-45d8-869a-440ae1ccd9d6 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:24.961933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a29f0c6-d1b9-45d8-869a-440ae1ccd9d6 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:24.962236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a29f0c6-d1b9-45d8-869a-440ae1ccd9d6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:24.974097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a29f0c6-d1b9-45d8-869a-440ae1ccd9d6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:24.975509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a29f0c6-d1b9-45d8-869a-440ae1ccd9d6 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-4a29f0c6-d1b9-45d8-869a-440ae1ccd9d6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:24.991194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:24.992121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:24.993010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:24.993010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:24.993010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:24.993010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:24.993010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 255/517] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:24.994359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a29f0c6-d1b9-45d8-869a-440ae1ccd9d6 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:25.008297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:25.014861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a3e4f6-673f-4338-86a2-571c330a15d1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:25.054981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae18f57-6533-4726-8a1f-595f3b576b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:25.055606 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae18f57-6533-4726-8a1f-595f3b576b6f None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:25.055890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae18f57-6533-4726-8a1f-595f3b576b6f None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:25.056167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae18f57-6533-4726-8a1f-595f3b576b6f None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:25.056897 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae18f57-6533-4726-8a1f-595f3b576b6f None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:25.057210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae18f57-6533-4726-8a1f-595f3b576b6f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:25.058581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae18f57-6533-4726-8a1f-595f3b576b6f None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:25.072191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 263/518] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:25.072883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae18f57-6533-4726-8a1f-595f3b576b6f None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:25.115065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a29f0c6-d1b9-45d8-869a-440ae1ccd9d6 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:25.148283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:25.157128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febf25a3-3d73-4946-a782-61065c4b538f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:25.158567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febf25a3-3d73-4946-a782-61065c4b538f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:25.158859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febf25a3-3d73-4946-a782-61065c4b538f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:25.159135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febf25a3-3d73-4946-a782-61065c4b538f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:25.173107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febf25a3-3d73-4946-a782-61065c4b538f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:25.175955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febf25a3-3d73-4946-a782-61065c4b538f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:25.184111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae18f57-6533-4726-8a1f-595f3b576b6f None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-febf25a3-3d73-4946-a782-61065c4b538f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:25.194040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:25.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:25.200816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:25.200816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:25.200816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:25.200816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:25.200816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 256/519] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:25.200816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febf25a3-3d73-4946-a782-61065c4b538f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:25.228675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503d685b-7142-473a-9ad8-7553c669cfd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:25.229224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503d685b-7142-473a-9ad8-7553c669cfd0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:25.229510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503d685b-7142-473a-9ad8-7553c669cfd0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:25.229794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503d685b-7142-473a-9ad8-7553c669cfd0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:25.271701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:25.307437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503d685b-7142-473a-9ad8-7553c669cfd0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:25.309027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503d685b-7142-473a-9ad8-7553c669cfd0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:25.317196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febf25a3-3d73-4946-a782-61065c4b538f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:25.352814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db7c111-5352-4ae9-b776-bd9e5d550b63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:25.353532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db7c111-5352-4ae9-b776-bd9e5d550b63 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:25.353892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db7c111-5352-4ae9-b776-bd9e5d550b63 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:25.354235 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db7c111-5352-4ae9-b776-bd9e5d550b63 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:25.355003 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db7c111-5352-4ae9-b776-bd9e5d550b63 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:25.355378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db7c111-5352-4ae9-b776-bd9e5d550b63 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:25.356882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db7c111-5352-4ae9-b776-bd9e5d550b63 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:25.373895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 257/520] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:25.373895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db7c111-5352-4ae9-b776-bd9e5d550b63 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:25.386942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:25.426916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 264/521] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:25.426916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503d685b-7142-473a-9ad8-7553c669cfd0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:25.486065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db7c111-5352-4ae9-b776-bd9e5d550b63 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:25.507428 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:25.524246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf18344-8bba-4ec0-b83f-b6d080919fd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:25.525001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf18344-8bba-4ec0-b83f-b6d080919fd3 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:25.525364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf18344-8bba-4ec0-b83f-b6d080919fd3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:25.525726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf18344-8bba-4ec0-b83f-b6d080919fd3 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:25.529499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503d685b-7142-473a-9ad8-7553c669cfd0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:25.534038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf18344-8bba-4ec0-b83f-b6d080919fd3 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:25.535516 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf18344-8bba-4ec0-b83f-b6d080919fd3 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-bdf18344-8bba-4ec0-b83f-b6d080919fd3 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:25.547642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:25.549588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 258/522] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:25 2022] GET /identity/v3/users/neutron => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:25.550488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf18344-8bba-4ec0-b83f-b6d080919fd3 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:25.573108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfb0f70-e28a-4887-b771-50d914a83aba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:25.573835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfb0f70-e28a-4887-b771-50d914a83aba None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:25.574217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfb0f70-e28a-4887-b771-50d914a83aba None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:25.574567 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cfb0f70-e28a-4887-b771-50d914a83aba None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:25.575280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfb0f70-e28a-4887-b771-50d914a83aba None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:25.575591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfb0f70-e28a-4887-b771-50d914a83aba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:25.577006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cfb0f70-e28a-4887-b771-50d914a83aba None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:25.593365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 265/523] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:25.593365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfb0f70-e28a-4887-b771-50d914a83aba None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:25.622423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:25.669818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf18344-8bba-4ec0-b83f-b6d080919fd3 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:25.707153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b88049-a728-4e6d-9251-d391f11a4bc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:25.707350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cfb0f70-e28a-4887-b771-50d914a83aba None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:25.708017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b88049-a728-4e6d-9251-d391f11a4bc4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:25.708405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b88049-a728-4e6d-9251-d391f11a4bc4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:25.708847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b88049-a728-4e6d-9251-d391f11a4bc4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:25.716453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b88049-a728-4e6d-9251-d391f11a4bc4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:25.718042 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b88049-a728-4e6d-9251-d391f11a4bc4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-24b88049-a728-4e6d-9251-d391f11a4bc4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:25.730605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:25.731480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:25.732453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:25.732453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:25.732453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:25.732453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:25.732453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:25.732453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:25.732453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:25.732453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:25.732453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 259/524] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:25.733312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b88049-a728-4e6d-9251-d391f11a4bc4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:25.747228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f39f1c1-600c-42d0-a401-483fd327d070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:25.747807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f39f1c1-600c-42d0-a401-483fd327d070 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:25.748092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f39f1c1-600c-42d0-a401-483fd327d070 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:25.748368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f39f1c1-600c-42d0-a401-483fd327d070 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:25.760736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f39f1c1-600c-42d0-a401-483fd327d070 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:25.762396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f39f1c1-600c-42d0-a401-483fd327d070 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:25.775990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-2f39f1c1-600c-42d0-a401-483fd327d070 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:25.779092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:25.779943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:25.781389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:25.781389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:25.781389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:25.781389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:25.781389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 266/525] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:25.781763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f39f1c1-600c-42d0-a401-483fd327d070 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:25.840351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b88049-a728-4e6d-9251-d391f11a4bc4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:25.870942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:25.876097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec86a0a-8505-45b6-a317-574f4176626e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:25.876830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec86a0a-8505-45b6-a317-574f4176626e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:25.877127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec86a0a-8505-45b6-a317-574f4176626e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:25.877404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec86a0a-8505-45b6-a317-574f4176626e None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:25.878208 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec86a0a-8505-45b6-a317-574f4176626e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:25.878522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec86a0a-8505-45b6-a317-574f4176626e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:25.879981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec86a0a-8505-45b6-a317-574f4176626e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:25.891171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f39f1c1-600c-42d0-a401-483fd327d070 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:25.898221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 260/526] 10.222.0.40 () {56 vars in 1139 bytes} [Fri Jul 29 04:34:25 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:25.901957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec86a0a-8505-45b6-a317-574f4176626e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:25.927972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f6319e-1ce5-4865-9812-631f969456f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:25.928693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f6319e-1ce5-4865-9812-631f969456f8 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:25.928951 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f6319e-1ce5-4865-9812-631f969456f8 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:25.929235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f6319e-1ce5-4865-9812-631f969456f8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:25.935978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f6319e-1ce5-4865-9812-631f969456f8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:25.937459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f6319e-1ce5-4865-9812-631f969456f8 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-89f6319e-1ce5-4865-9812-631f969456f8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:25.950183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:25.951036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:25.951887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:25.951887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:25.951887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:25.951887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:25.951887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:25.951887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:25.951887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:25.951887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:25.951887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 267/527] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:25.952422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f6319e-1ce5-4865-9812-631f969456f8 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:26.017594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:26.022076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec86a0a-8505-45b6-a317-574f4176626e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:26.058285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ecaaaa-062e-4df2-a461-5085e9628714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:26.058909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ecaaaa-062e-4df2-a461-5085e9628714 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:26.059193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ecaaaa-062e-4df2-a461-5085e9628714 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:26.059469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ecaaaa-062e-4df2-a461-5085e9628714 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:26.060218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ecaaaa-062e-4df2-a461-5085e9628714 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:26.060526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ecaaaa-062e-4df2-a461-5085e9628714 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:26.062008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ecaaaa-062e-4df2-a461-5085e9628714 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:26.079877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 261/528] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:26.085618 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ecaaaa-062e-4df2-a461-5085e9628714 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:26.086822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f6319e-1ce5-4865-9812-631f969456f8 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:26.122007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3dfad1-e367-4323-a592-26825f0c123e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:26.122745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3dfad1-e367-4323-a592-26825f0c123e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:26.123040 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3dfad1-e367-4323-a592-26825f0c123e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:26.123321 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3dfad1-e367-4323-a592-26825f0c123e None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:26.124018 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3dfad1-e367-4323-a592-26825f0c123e None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:26.124332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3dfad1-e367-4323-a592-26825f0c123e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:26.125755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3dfad1-e367-4323-a592-26825f0c123e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:26.138875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 268/529] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:26.139734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3dfad1-e367-4323-a592-26825f0c123e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:26.142889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:26.201209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ecaaaa-062e-4df2-a461-5085e9628714 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:26.237947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99e981f-4d0f-480b-8e9f-31dbc121bb04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:26.238712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99e981f-4d0f-480b-8e9f-31dbc121bb04 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:26.239194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99e981f-4d0f-480b-8e9f-31dbc121bb04 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:26.239677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99e981f-4d0f-480b-8e9f-31dbc121bb04 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:26.246066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3dfad1-e367-4323-a592-26825f0c123e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:26.267860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:26.285686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb36c08-bab7-44f2-ad2d-687825880acb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:26.286314 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb36c08-bab7-44f2-ad2d-687825880acb None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:26.286601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb36c08-bab7-44f2-ad2d-687825880acb None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:26.286887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb36c08-bab7-44f2-ad2d-687825880acb None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:26.287648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb36c08-bab7-44f2-ad2d-687825880acb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:26.287965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb36c08-bab7-44f2-ad2d-687825880acb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:26.289455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb36c08-bab7-44f2-ad2d-687825880acb None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:26.312930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 269/530] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:26.312930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb36c08-bab7-44f2-ad2d-687825880acb None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:26.317887 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99e981f-4d0f-480b-8e9f-31dbc121bb04 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:26.319580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99e981f-4d0f-480b-8e9f-31dbc121bb04 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:26.374188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:26.414622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb36c08-bab7-44f2-ad2d-687825880acb None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:26.426249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 262/531] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:26.427257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99e981f-4d0f-480b-8e9f-31dbc121bb04 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:26.454005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0408bd63-bd96-4e0b-951c-3cc3eedbede2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:26.454708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0408bd63-bd96-4e0b-951c-3cc3eedbede2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:26.454999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0408bd63-bd96-4e0b-951c-3cc3eedbede2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:26.455396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0408bd63-bd96-4e0b-951c-3cc3eedbede2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:26.456200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0408bd63-bd96-4e0b-951c-3cc3eedbede2 None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=58c33a72a871432b88da4f0bf1bc514c) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:26.456521 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0408bd63-bd96-4e0b-951c-3cc3eedbede2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:26.457993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0408bd63-bd96-4e0b-951c-3cc3eedbede2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:26.463498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 270/532] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:26 2022] GET /identity/v3/role_assignments?user.id=58c33a72a871432b88da4f0bf1bc514c&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:26.464282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0408bd63-bd96-4e0b-951c-3cc3eedbede2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:26.511932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:26.568645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99e981f-4d0f-480b-8e9f-31dbc121bb04 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:26.593415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0408bd63-bd96-4e0b-951c-3cc3eedbede2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:26.608328 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5b7fd6-9bfd-461f-8afc-eba248f21485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:26.609030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5b7fd6-9bfd-461f-8afc-eba248f21485 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:26.609326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5b7fd6-9bfd-461f-8afc-eba248f21485 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:26.609860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5b7fd6-9bfd-461f-8afc-eba248f21485 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:26.617867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5b7fd6-9bfd-461f-8afc-eba248f21485 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:26.619567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5b7fd6-9bfd-461f-8afc-eba248f21485 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:26.630960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-ca5b7fd6-9bfd-461f-8afc-eba248f21485 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:26.633297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:26.634146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:26.635025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:26.635025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:26.635025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:26.635025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:26.635025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:26.635025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:26.635025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:26.635025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:26.635025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 263/533] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:26.635476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5b7fd6-9bfd-461f-8afc-eba248f21485 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:26.646732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d334a1a-9997-4d6b-b3b9-80941498af7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:26.647605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d334a1a-9997-4d6b-b3b9-80941498af7a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:26.648040 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d334a1a-9997-4d6b-b3b9-80941498af7a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:26.648468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d334a1a-9997-4d6b-b3b9-80941498af7a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:26.649882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d334a1a-9997-4d6b-b3b9-80941498af7a None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=de49172e2a054096b319a7e4349ecaea) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:26.650337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d334a1a-9997-4d6b-b3b9-80941498af7a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:26.652562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d334a1a-9997-4d6b-b3b9-80941498af7a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:26.660715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 271/534] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:26 2022] GET /identity/v3/role_assignments?user.id=de49172e2a054096b319a7e4349ecaea&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 259 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:26.661760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d334a1a-9997-4d6b-b3b9-80941498af7a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:26.753628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5b7fd6-9bfd-461f-8afc-eba248f21485 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:26.773197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:26.792291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef77f0d-1ce9-4211-b70d-7279309879d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:26.793326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef77f0d-1ce9-4211-b70d-7279309879d7 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:26.793549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef77f0d-1ce9-4211-b70d-7279309879d7 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:26.793930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef77f0d-1ce9-4211-b70d-7279309879d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:26.804262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d334a1a-9997-4d6b-b3b9-80941498af7a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:26.807580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef77f0d-1ce9-4211-b70d-7279309879d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:26.808500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef77f0d-1ce9-4211-b70d-7279309879d7 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-bef77f0d-1ce9-4211-b70d-7279309879d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:26.824281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:26.825207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:26.826040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:26.826040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:26.826040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:26.826040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:26.826248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 264/535] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:26.837810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef77f0d-1ce9-4211-b70d-7279309879d7 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:26.846457 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62ad205-4479-4885-8ac6-037b175fff95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:26.847504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62ad205-4479-4885-8ac6-037b175fff95 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:26.848013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62ad205-4479-4885-8ac6-037b175fff95 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:26.848493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62ad205-4479-4885-8ac6-037b175fff95 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:26.849502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62ad205-4479-4885-8ac6-037b175fff95 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:26.850054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62ad205-4479-4885-8ac6-037b175fff95 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:26.851817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62ad205-4479-4885-8ac6-037b175fff95 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:26.867026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 272/536] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:26.873412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62ad205-4479-4885-8ac6-037b175fff95 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:26.933997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:26.990545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62ad205-4479-4885-8ac6-037b175fff95 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:27.010909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef77f0d-1ce9-4211-b70d-7279309879d7 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:27.029046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c90df54-660e-4c06-9489-da0083ce68ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:27.030039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90df54-660e-4c06-9489-da0083ce68ad None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:27.030527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90df54-660e-4c06-9489-da0083ce68ad None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:27.031030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c90df54-660e-4c06-9489-da0083ce68ad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:27.032216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c90df54-660e-4c06-9489-da0083ce68ad None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=3749dc021aac478f88072ced48f4cee4) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:27.033345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c90df54-660e-4c06-9489-da0083ce68ad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:27.035071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c90df54-660e-4c06-9489-da0083ce68ad None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:27.043124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 273/537] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:26 2022] GET /identity/v3/role_assignments?user.id=3749dc021aac478f88072ced48f4cee4&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:27.050271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def043fe-8c67-4b33-b0a7-d48edec18cc3 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:27.050558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def043fe-8c67-4b33-b0a7-d48edec18cc3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:27.050848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def043fe-8c67-4b33-b0a7-d48edec18cc3 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:27.052227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 274/538] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:27.055463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f670ba-fd3b-45a3-b460-a1efe381b21e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:27.056107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f670ba-fd3b-45a3-b460-a1efe381b21e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:27.056553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f670ba-fd3b-45a3-b460-a1efe381b21e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:27.056778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f670ba-fd3b-45a3-b460-a1efe381b21e None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:27.057505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f670ba-fd3b-45a3-b460-a1efe381b21e None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:27.057816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f670ba-fd3b-45a3-b460-a1efe381b21e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:27.059277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f670ba-fd3b-45a3-b460-a1efe381b21e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:27.062488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091d529f-2e30-47e7-ae25-1c1549c09782 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:27.062783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091d529f-2e30-47e7-ae25-1c1549c09782 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:27.063065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091d529f-2e30-47e7-ae25-1c1549c09782 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:27.091950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 265/539] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:27.099987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f670ba-fd3b-45a3-b460-a1efe381b21e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:27.109215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-091d529f-2e30-47e7-ae25-1c1549c09782 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:27.110486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-091d529f-2e30-47e7-ae25-1c1549c09782 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:27.189594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:27.217171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 275/540] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:27.248300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091d529f-2e30-47e7-ae25-1c1549c09782 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:27.278989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f670ba-fd3b-45a3-b460-a1efe381b21e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:27.316428 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f948423e-7d97-4562-b7f8-f2d7b02a1e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:27.317382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f948423e-7d97-4562-b7f8-f2d7b02a1e43 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:27.317823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f948423e-7d97-4562-b7f8-f2d7b02a1e43 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:27.318242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f948423e-7d97-4562-b7f8-f2d7b02a1e43 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:27.325973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f948423e-7d97-4562-b7f8-f2d7b02a1e43 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:27.326427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:27.331703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f948423e-7d97-4562-b7f8-f2d7b02a1e43 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-f948423e-7d97-4562-b7f8-f2d7b02a1e43 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:27.346513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:27.347568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:27.348575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:27.348575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:27.348575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:27.348575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:27.348575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:27.348575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:27.348575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:27.348575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:27.350701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 266/541] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:27.357016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f948423e-7d97-4562-b7f8-f2d7b02a1e43 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:27.432682 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091d529f-2e30-47e7-ae25-1c1549c09782 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:27.491758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6531789-00f6-4296-b2f2-a3d0737e91b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:27.492665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6531789-00f6-4296-b2f2-a3d0737e91b2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:27.493088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6531789-00f6-4296-b2f2-a3d0737e91b2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:27.493503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6531789-00f6-4296-b2f2-a3d0737e91b2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:27.509299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6531789-00f6-4296-b2f2-a3d0737e91b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-b6531789-00f6-4296-b2f2-a3d0737e91b2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:27.521883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:27.522970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:27.523874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:27.525864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 276/542] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:27.529354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:27.532317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6531789-00f6-4296-b2f2-a3d0737e91b2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:27.551971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f948423e-7d97-4562-b7f8-f2d7b02a1e43 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:27.594505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bac1dc1-11e9-41c1-84db-ae92f75c4b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:27.595549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bac1dc1-11e9-41c1-84db-ae92f75c4b46 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:27.596058 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bac1dc1-11e9-41c1-84db-ae92f75c4b46 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:27.596476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bac1dc1-11e9-41c1-84db-ae92f75c4b46 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:27.597472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bac1dc1-11e9-41c1-84db-ae92f75c4b46 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:27.597931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bac1dc1-11e9-41c1-84db-ae92f75c4b46 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:27.599677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bac1dc1-11e9-41c1-84db-ae92f75c4b46 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:27.613665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 267/543] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:27.619503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bac1dc1-11e9-41c1-84db-ae92f75c4b46 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:27.727263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bac1dc1-11e9-41c1-84db-ae92f75c4b46 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:27.759246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6531789-00f6-4296-b2f2-a3d0737e91b2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:27.761788 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:27.766669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda6d63e-9ed5-433b-a976-d6ec4fa0b0e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:27.768394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda6d63e-9ed5-433b-a976-d6ec4fa0b0e8 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:27.768878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda6d63e-9ed5-433b-a976-d6ec4fa0b0e8 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:27.769299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda6d63e-9ed5-433b-a976-d6ec4fa0b0e8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:27.770261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda6d63e-9ed5-433b-a976-d6ec4fa0b0e8 None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=f6726ef0c4184040a18a76898bb738d1) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:27.770793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda6d63e-9ed5-433b-a976-d6ec4fa0b0e8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:27.772499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda6d63e-9ed5-433b-a976-d6ec4fa0b0e8 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:27.779178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 268/544] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:27 2022] GET /identity/v3/role_assignments?user.id=f6726ef0c4184040a18a76898bb738d1&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:27.796959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fbd21d-d2a7-4939-848c-09156ea45ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:27.797761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fbd21d-d2a7-4939-848c-09156ea45ca4 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:27.798500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fbd21d-d2a7-4939-848c-09156ea45ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:27.799003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fbd21d-d2a7-4939-848c-09156ea45ca4 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:27.799976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fbd21d-d2a7-4939-848c-09156ea45ca4 None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:27.800394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fbd21d-d2a7-4939-848c-09156ea45ca4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:27.801965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fbd21d-d2a7-4939-848c-09156ea45ca4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:27.811863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 277/545] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:27.818555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda6d63e-9ed5-433b-a976-d6ec4fa0b0e8 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:27.844757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:27.950890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda6d63e-9ed5-433b-a976-d6ec4fa0b0e8 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:27.989448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5338cd12-6ae4-43ed-9f51-aa0eb5f98b64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:27.990368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5338cd12-6ae4-43ed-9f51-aa0eb5f98b64 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:27.990792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5338cd12-6ae4-43ed-9f51-aa0eb5f98b64 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:27.991209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5338cd12-6ae4-43ed-9f51-aa0eb5f98b64 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:28.004419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5338cd12-6ae4-43ed-9f51-aa0eb5f98b64 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:28.006193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5338cd12-6ae4-43ed-9f51-aa0eb5f98b64 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-5338cd12-6ae4-43ed-9f51-aa0eb5f98b64 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:28.022057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:28.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:28.024157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:28.024157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:28.024157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:28.024157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:28.026119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 269/546] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:28.034444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fbd21d-d2a7-4939-848c-09156ea45ca4 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:28.163723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fbd21d-d2a7-4939-848c-09156ea45ca4 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:28.197782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2375b810-8512-4711-b5f1-4db1e45471e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:28.198409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2375b810-8512-4711-b5f1-4db1e45471e8 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:28.198756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2375b810-8512-4711-b5f1-4db1e45471e8 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:28.199044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2375b810-8512-4711-b5f1-4db1e45471e8 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:28.199742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2375b810-8512-4711-b5f1-4db1e45471e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:28.200054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2375b810-8512-4711-b5f1-4db1e45471e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:28.201508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2375b810-8512-4711-b5f1-4db1e45471e8 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:28.214727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 278/547] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:28.221116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5338cd12-6ae4-43ed-9f51-aa0eb5f98b64 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:28.325839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:28.366102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5338cd12-6ae4-43ed-9f51-aa0eb5f98b64 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:28.402074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c371c5c4-15ef-4c7b-8d3c-eed61363a51f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:28.403085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c371c5c4-15ef-4c7b-8d3c-eed61363a51f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:28.403566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c371c5c4-15ef-4c7b-8d3c-eed61363a51f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:28.404063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c371c5c4-15ef-4c7b-8d3c-eed61363a51f None admin] PATH_INFO: `/v3/users/swift` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:28.412780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c371c5c4-15ef-4c7b-8d3c-eed61363a51f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:28.414628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c371c5c4-15ef-4c7b-8d3c-eed61363a51f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c371c5c4-15ef-4c7b-8d3c-eed61363a51f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:28.428283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:28.429342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:28.430681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:28.430681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:28.430681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:28.430681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:28.430681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:28.430681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:28.430681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:28.430681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:28.430681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:28.430681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:28.433532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 270/548] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:28 2022] GET /identity/v3/users/swift => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:28.442038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2375b810-8512-4711-b5f1-4db1e45471e8 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:28.486130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:28.572161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2375b810-8512-4711-b5f1-4db1e45471e8 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:28.608359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a600f63f-1c9e-47db-b39a-5355868710d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:28.609023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a600f63f-1c9e-47db-b39a-5355868710d9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:28.609580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a600f63f-1c9e-47db-b39a-5355868710d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:28.609863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a600f63f-1c9e-47db-b39a-5355868710d9 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:28.610734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a600f63f-1c9e-47db-b39a-5355868710d9 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:28.611050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a600f63f-1c9e-47db-b39a-5355868710d9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:28.612595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a600f63f-1c9e-47db-b39a-5355868710d9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:28.631291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 279/549] 10.222.0.40 () {56 vars in 1135 bytes} [Fri Jul 29 04:34:28 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:28.636465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c371c5c4-15ef-4c7b-8d3c-eed61363a51f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:28.747969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c371c5c4-15ef-4c7b-8d3c-eed61363a51f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:28.775316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:28.792088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ca6aa2-063f-41ef-b62f-d84a54ccffbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:28.793060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ca6aa2-063f-41ef-b62f-d84a54ccffbc None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:28.793570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ca6aa2-063f-41ef-b62f-d84a54ccffbc None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:28.794045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ca6aa2-063f-41ef-b62f-d84a54ccffbc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:28.887914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ca6aa2-063f-41ef-b62f-d84a54ccffbc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:28.889864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ca6aa2-063f-41ef-b62f-d84a54ccffbc None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:28.994486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 271/550] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:28.999840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a600f63f-1c9e-47db-b39a-5355868710d9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:29.115006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0178edf7-4be4-468e-831a-07cbd2ec6c47 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.115311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0178edf7-4be4-468e-831a-07cbd2ec6c47 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.115578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0178edf7-4be4-468e-831a-07cbd2ec6c47 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.117222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 272/551] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:29.123616 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b110c50-f9a5-4b9e-851b-e14fed1b81d6 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.123914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b110c50-f9a5-4b9e-851b-e14fed1b81d6 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.124200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b110c50-f9a5-4b9e-851b-e14fed1b81d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.142885 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:29.148881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a600f63f-1c9e-47db-b39a-5355868710d9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:29.158692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4b110c50-f9a5-4b9e-851b-e14fed1b81d6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:29.159926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4b110c50-f9a5-4b9e-851b-e14fed1b81d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:29.199573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:29.202331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656275dd-f678-427e-999c-04562680a0b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:29.203054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656275dd-f678-427e-999c-04562680a0b1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.203341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656275dd-f678-427e-999c-04562680a0b1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.203628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656275dd-f678-427e-999c-04562680a0b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.224203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656275dd-f678-427e-999c-04562680a0b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:29.226280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656275dd-f678-427e-999c-04562680a0b1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-656275dd-f678-427e-999c-04562680a0b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:29.256685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:29.257648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:29.258497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:29.258497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:29.258497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:29.258497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:29.258497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 280/552] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:29.265637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656275dd-f678-427e-999c-04562680a0b1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:29.292787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 273/553] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:29.294725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd0fd5-868a-429e-9ee2-4388647d6770 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.295020 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd0fd5-868a-429e-9ee2-4388647d6770 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.295302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd0fd5-868a-429e-9ee2-4388647d6770 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.297458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 274/554] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:29.305926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecda2a1-99fd-470d-88c2-c683ead29d53 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.306304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecda2a1-99fd-470d-88c2-c683ead29d53 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.306659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecda2a1-99fd-470d-88c2-c683ead29d53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.331501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-5ecda2a1-99fd-470d-88c2-c683ead29d53 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:29.333031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5ecda2a1-99fd-470d-88c2-c683ead29d53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:29.398648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656275dd-f678-427e-999c-04562680a0b1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:29.420727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 275/555] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:34:29.421789 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecda2a1-99fd-470d-88c2-c683ead29d53 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:29.440309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ef07dc-8d41-40dd-8da1-5d33576f68a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:29.440964 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ef07dc-8d41-40dd-8da1-5d33576f68a8 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.441250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ef07dc-8d41-40dd-8da1-5d33576f68a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.441531 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ef07dc-8d41-40dd-8da1-5d33576f68a8 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.442226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ef07dc-8d41-40dd-8da1-5d33576f68a8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:29.443011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ef07dc-8d41-40dd-8da1-5d33576f68a8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:29.444901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ef07dc-8d41-40dd-8da1-5d33576f68a8 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:29.447479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:29.461759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 281/556] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:29.462684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ef07dc-8d41-40dd-8da1-5d33576f68a8 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:29.534421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:29.552820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecda2a1-99fd-470d-88c2-c683ead29d53 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:29.589476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b80baff-3a73-4561-9b99-44232b32f679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:29.590126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b80baff-3a73-4561-9b99-44232b32f679 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.590416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b80baff-3a73-4561-9b99-44232b32f679 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.590699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b80baff-3a73-4561-9b99-44232b32f679 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.597929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ef07dc-8d41-40dd-8da1-5d33576f68a8 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:29.602524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b80baff-3a73-4561-9b99-44232b32f679 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-8b80baff-3a73-4561-9b99-44232b32f679 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:29.607690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:29.608619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:29.610029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 276/557] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:29.611415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b80baff-3a73-4561-9b99-44232b32f679 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:29.646311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a7f17e-838d-49ff-acc3-d68698b6cb12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:29.647270 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a7f17e-838d-49ff-acc3-d68698b6cb12 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.647563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a7f17e-838d-49ff-acc3-d68698b6cb12 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.647847 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a7f17e-838d-49ff-acc3-d68698b6cb12 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.666485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a7f17e-838d-49ff-acc3-d68698b6cb12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-96a7f17e-838d-49ff-acc3-d68698b6cb12 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:29.671824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:29.673844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:29.674871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:29.676976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 282/558] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:29.678045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a7f17e-838d-49ff-acc3-d68698b6cb12 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:29.728784 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b80baff-3a73-4561-9b99-44232b32f679 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:29.745703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:29.765753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825d2742-bc96-4aaf-88de-225183bad3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:29.766408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825d2742-bc96-4aaf-88de-225183bad3e1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.766682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825d2742-bc96-4aaf-88de-225183bad3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.767529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825d2742-bc96-4aaf-88de-225183bad3e1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.775463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825d2742-bc96-4aaf-88de-225183bad3e1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:29.776945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825d2742-bc96-4aaf-88de-225183bad3e1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-825d2742-bc96-4aaf-88de-225183bad3e1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:29.789112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:29.790017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:29.791007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:29.791007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:29.791007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:29.791007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:29.791007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:29.791007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:29.791007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:29.791007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:29.791007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 277/559] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:29.791971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2733ddd0-5ec4-4e5d-8056-d860362f89fa None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.791971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2733ddd0-5ec4-4e5d-8056-d860362f89fa None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.791971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2733ddd0-5ec4-4e5d-8056-d860362f89fa None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.793273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 278/560] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:29 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:29.797705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2733ddd0-5ec4-4e5d-8056-d860362f89fa None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:29.799138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a7f17e-838d-49ff-acc3-d68698b6cb12 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:29.808322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:29.837755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a199c9a6-4dab-4f4e-beed-6579ff1e498a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:29.838617 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a199c9a6-4dab-4f4e-beed-6579ff1e498a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.838903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a199c9a6-4dab-4f4e-beed-6579ff1e498a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.839185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a199c9a6-4dab-4f4e-beed-6579ff1e498a None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.839902 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a199c9a6-4dab-4f4e-beed-6579ff1e498a None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:29.840207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a199c9a6-4dab-4f4e-beed-6579ff1e498a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:29.841630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a199c9a6-4dab-4f4e-beed-6579ff1e498a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:29.850907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 283/561] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:29.855921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a199c9a6-4dab-4f4e-beed-6579ff1e498a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:29.921782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2733ddd0-5ec4-4e5d-8056-d860362f89fa None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:29.958537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5410a5fb-d053-4a35-bfed-5ae68377bed1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:29.959154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5410a5fb-d053-4a35-bfed-5ae68377bed1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.959435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5410a5fb-d053-4a35-bfed-5ae68377bed1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.959713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5410a5fb-d053-4a35-bfed-5ae68377bed1 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.960550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5410a5fb-d053-4a35-bfed-5ae68377bed1 None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:29.960885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5410a5fb-d053-4a35-bfed-5ae68377bed1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:29.962250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5410a5fb-d053-4a35-bfed-5ae68377bed1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:29.975216 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 279/562] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:29.975216 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2966c-c23f-4e43-b6fb-f00247785000 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:29.975216 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2966c-c23f-4e43-b6fb-f00247785000 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:29.975528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a199c9a6-4dab-4f4e-beed-6579ff1e498a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:29.975605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2966c-c23f-4e43-b6fb-f00247785000 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:29.991770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:29.992381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f4e2966c-c23f-4e43-b6fb-f00247785000 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:29.993790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f4e2966c-c23f-4e43-b6fb-f00247785000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:30.009789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6d05c8-b014-478e-a4f3-d879a92d6ddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:30.010496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6d05c8-b014-478e-a4f3-d879a92d6ddf None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.010786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6d05c8-b014-478e-a4f3-d879a92d6ddf None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.011060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6d05c8-b014-478e-a4f3-d879a92d6ddf None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.011821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6d05c8-b014-478e-a4f3-d879a92d6ddf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:30.012131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6d05c8-b014-478e-a4f3-d879a92d6ddf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:30.013601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6d05c8-b014-478e-a4f3-d879a92d6ddf None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:30.030613 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 284/563] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:30.030613 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6d05c8-b014-478e-a4f3-d879a92d6ddf None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.063124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:30.080638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 280/564] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:30.080638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2966c-c23f-4e43-b6fb-f00247785000 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.144298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6d05c8-b014-478e-a4f3-d879a92d6ddf None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:30.179747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acba3cd-976c-4535-b395-5f04020c60de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:30.180474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acba3cd-976c-4535-b395-5f04020c60de None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.180729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acba3cd-976c-4535-b395-5f04020c60de None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.181007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acba3cd-976c-4535-b395-5f04020c60de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.185281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2966c-c23f-4e43-b6fb-f00247785000 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:30.192096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acba3cd-976c-4535-b395-5f04020c60de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:30.193584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acba3cd-976c-4535-b395-5f04020c60de None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-0acba3cd-976c-4535-b395-5f04020c60de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:30.208612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:30.210133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:30.211209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:30.211209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:30.211209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:30.211209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:30.214596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 285/565] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:30.215414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acba3cd-976c-4535-b395-5f04020c60de None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.225545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef447b8-90d8-421e-acf4-5349a994e6d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:30.226195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef447b8-90d8-421e-acf4-5349a994e6d3 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.226491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef447b8-90d8-421e-acf4-5349a994e6d3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.226767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef447b8-90d8-421e-acf4-5349a994e6d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.236966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef447b8-90d8-421e-acf4-5349a994e6d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:30.238438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef447b8-90d8-421e-acf4-5349a994e6d3 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:30.246008 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-bef447b8-90d8-421e-acf4-5349a994e6d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:30.253881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:30.254896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:30.255776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:30.255776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:30.255776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:30.255776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:30.255776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 281/566] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:30.255776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef447b8-90d8-421e-acf4-5349a994e6d3 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.307921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:30.316977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acba3cd-976c-4535-b395-5f04020c60de None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:30.349676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d669482b-2507-44c6-96f6-ab85ccadfd66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:30.350293 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d669482b-2507-44c6-96f6-ab85ccadfd66 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.350580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d669482b-2507-44c6-96f6-ab85ccadfd66 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.350859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d669482b-2507-44c6-96f6-ab85ccadfd66 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.351648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d669482b-2507-44c6-96f6-ab85ccadfd66 None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=bbfe92babca048d882cb59cb29b3ed97) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:30.351962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d669482b-2507-44c6-96f6-ab85ccadfd66 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:30.353461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d669482b-2507-44c6-96f6-ab85ccadfd66 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:30.358601 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef447b8-90d8-421e-acf4-5349a994e6d3 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:30.359139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 286/567] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/role_assignments?user.id=bbfe92babca048d882cb59cb29b3ed97&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:30.360364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe2d010-289a-494c-84d3-4c562b0a8eba None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.361238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe2d010-289a-494c-84d3-4c562b0a8eba None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.362284 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe2d010-289a-494c-84d3-4c562b0a8eba None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.362926 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 287/568] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:30.363613 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe2d010-289a-494c-84d3-4c562b0a8eba None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.399810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add6808f-88e9-4a08-b7b1-9113ebb5ad28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:30.400508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add6808f-88e9-4a08-b7b1-9113ebb5ad28 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.400824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add6808f-88e9-4a08-b7b1-9113ebb5ad28 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.401106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add6808f-88e9-4a08-b7b1-9113ebb5ad28 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.413330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add6808f-88e9-4a08-b7b1-9113ebb5ad28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-add6808f-88e9-4a08-b7b1-9113ebb5ad28 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:30.419069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:30.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:30.421252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 282/569] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:30.422031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add6808f-88e9-4a08-b7b1-9113ebb5ad28 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.490788 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:30.501385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe2d010-289a-494c-84d3-4c562b0a8eba None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:30.540991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add6808f-88e9-4a08-b7b1-9113ebb5ad28 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:30.541731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e65bb2f-e227-4f53-a28c-727d7559780c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:30.542468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e65bb2f-e227-4f53-a28c-727d7559780c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.542825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e65bb2f-e227-4f53-a28c-727d7559780c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.543115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e65bb2f-e227-4f53-a28c-727d7559780c None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.543827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e65bb2f-e227-4f53-a28c-727d7559780c None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:30.544158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e65bb2f-e227-4f53-a28c-727d7559780c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:30.545556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e65bb2f-e227-4f53-a28c-727d7559780c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:30.553074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:30.560673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 288/570] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:30.566174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ccf6a-bada-4b09-a4b4-7bb178f48bcb None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.566477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ccf6a-bada-4b09-a4b4-7bb178f48bcb None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.566848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ccf6a-bada-4b09-a4b4-7bb178f48bcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.580256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0beb9142-aac9-4923-8adb-6a533e3f960d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:30.580939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0beb9142-aac9-4923-8adb-6a533e3f960d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.581294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0beb9142-aac9-4923-8adb-6a533e3f960d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.581645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0beb9142-aac9-4923-8adb-6a533e3f960d None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.582420 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0beb9142-aac9-4923-8adb-6a533e3f960d None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:30.582804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0beb9142-aac9-4923-8adb-6a533e3f960d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:30.584266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0beb9142-aac9-4923-8adb-6a533e3f960d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:30.585616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-595ccf6a-bada-4b09-a4b4-7bb178f48bcb None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:30.586832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-595ccf6a-bada-4b09-a4b4-7bb178f48bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:30.597226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 283/571] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:30.598355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0beb9142-aac9-4923-8adb-6a533e3f960d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.672310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 289/572] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:30.674405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595ccf6a-bada-4b09-a4b4-7bb178f48bcb None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.708258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0beb9142-aac9-4923-8adb-6a533e3f960d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:30.746794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc576f97-0c80-41a8-b400-3a35ec287682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:30.747674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc576f97-0c80-41a8-b400-3a35ec287682 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.748177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc576f97-0c80-41a8-b400-3a35ec287682 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.748552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc576f97-0c80-41a8-b400-3a35ec287682 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.749408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc576f97-0c80-41a8-b400-3a35ec287682 None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:30.749822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc576f97-0c80-41a8-b400-3a35ec287682 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:30.752384 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc576f97-0c80-41a8-b400-3a35ec287682 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:30.765959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:30.766827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 284/573] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:30.768285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc576f97-0c80-41a8-b400-3a35ec287682 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.778923 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595ccf6a-bada-4b09-a4b4-7bb178f48bcb None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:30.803871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:30.820719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f64d3d1-2071-44ad-aa6b-1db7761e6671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:30.821380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f64d3d1-2071-44ad-aa6b-1db7761e6671 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.821669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f64d3d1-2071-44ad-aa6b-1db7761e6671 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.821950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f64d3d1-2071-44ad-aa6b-1db7761e6671 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.830451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f64d3d1-2071-44ad-aa6b-1db7761e6671 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:30.831891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f64d3d1-2071-44ad-aa6b-1db7761e6671 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-6f64d3d1-2071-44ad-aa6b-1db7761e6671 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:30.843755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:30.844798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:30.846253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 290/574] 10.222.0.40 () {56 vars in 1093 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/users/nova => generated 82 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:30.846839 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f64d3d1-2071-44ad-aa6b-1db7761e6671 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.883483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc576f97-0c80-41a8-b400-3a35ec287682 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:30.921381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20077da4-dec7-4fcf-a57e-117d5da17d8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:30.922035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20077da4-dec7-4fcf-a57e-117d5da17d8b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:30.922330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20077da4-dec7-4fcf-a57e-117d5da17d8b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:30.922611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20077da4-dec7-4fcf-a57e-117d5da17d8b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:30.931725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20077da4-dec7-4fcf-a57e-117d5da17d8b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:30.933261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20077da4-dec7-4fcf-a57e-117d5da17d8b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-20077da4-dec7-4fcf-a57e-117d5da17d8b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:30.945377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:30.946335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 285/575] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/users/cinder => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:30.947279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20077da4-dec7-4fcf-a57e-117d5da17d8b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:30.963190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f64d3d1-2071-44ad-aa6b-1db7761e6671 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.000030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0d83e2-faeb-4b86-be6e-bf4d76d39279 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.001046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0d83e2-faeb-4b86-be6e-bf4d76d39279 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.001635 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0d83e2-faeb-4b86-be6e-bf4d76d39279 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.002146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0d83e2-faeb-4b86-be6e-bf4d76d39279 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.012538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0d83e2-faeb-4b86-be6e-bf4d76d39279 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-3e0d83e2-faeb-4b86-be6e-bf4d76d39279 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:31.018103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:31.019188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:31.020042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 291/576] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:31.020728 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0d83e2-faeb-4b86-be6e-bf4d76d39279 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.028707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:31.053828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:31.066045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20077da4-dec7-4fcf-a57e-117d5da17d8b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.104400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd3e32-e4d2-4427-8bbb-0c5ee9132801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.105001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd3e32-e4d2-4427-8bbb-0c5ee9132801 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.105290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd3e32-e4d2-4427-8bbb-0c5ee9132801 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.105571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd3e32-e4d2-4427-8bbb-0c5ee9132801 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.117851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cd3e32-e4d2-4427-8bbb-0c5ee9132801 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:31.119312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cd3e32-e4d2-4427-8bbb-0c5ee9132801 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-65cd3e32-e4d2-4427-8bbb-0c5ee9132801 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:31.133619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:31.134505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:31.135371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:31.135371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:31.135371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:31.135371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:31.135371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 286/577] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:31.135645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd3e32-e4d2-4427-8bbb-0c5ee9132801 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.146115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0d83e2-faeb-4b86-be6e-bf4d76d39279 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.182251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce469cd9-ddda-45d6-934a-3fe4496c21f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.182971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce469cd9-ddda-45d6-934a-3fe4496c21f4 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.183257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce469cd9-ddda-45d6-934a-3fe4496c21f4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.183539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce469cd9-ddda-45d6-934a-3fe4496c21f4 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.184335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce469cd9-ddda-45d6-934a-3fe4496c21f4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:31.184678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce469cd9-ddda-45d6-934a-3fe4496c21f4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:31.186120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce469cd9-ddda-45d6-934a-3fe4496c21f4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:31.208694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 292/578] 10.222.0.40 () {56 vars in 1133 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:31.208694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce469cd9-ddda-45d6-934a-3fe4496c21f4 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.245634 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd3e32-e4d2-4427-8bbb-0c5ee9132801 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.272795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:31.285766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae31ae-d819-4d21-a3e5-f18bddc747ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.286624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae31ae-d819-4d21-a3e5-f18bddc747ff None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.287087 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae31ae-d819-4d21-a3e5-f18bddc747ff None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.287515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ae31ae-d819-4d21-a3e5-f18bddc747ff None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.288582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae31ae-d819-4d21-a3e5-f18bddc747ff None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:31.289068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae31ae-d819-4d21-a3e5-f18bddc747ff None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:31.290642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99ae31ae-d819-4d21-a3e5-f18bddc747ff None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:31.309307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 287/579] 10.222.0.40 () {56 vars in 1137 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:31.310332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae31ae-d819-4d21-a3e5-f18bddc747ff None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.312909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:31.319253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce469cd9-ddda-45d6-934a-3fe4496c21f4 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.357779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29bcfcd-dffd-48b8-873d-a032a6f94177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.358642 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29bcfcd-dffd-48b8-873d-a032a6f94177 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.359451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29bcfcd-dffd-48b8-873d-a032a6f94177 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.359890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29bcfcd-dffd-48b8-873d-a032a6f94177 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.361636 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29bcfcd-dffd-48b8-873d-a032a6f94177 None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:31.362382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29bcfcd-dffd-48b8-873d-a032a6f94177 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:31.364086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29bcfcd-dffd-48b8-873d-a032a6f94177 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:31.373543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 293/580] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:31.374340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29bcfcd-dffd-48b8-873d-a032a6f94177 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.427995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ae31ae-d819-4d21-a3e5-f18bddc747ff None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.467878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193f933b-2d50-408a-9e4d-ce2768399c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.468515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193f933b-2d50-408a-9e4d-ce2768399c1d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.468832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193f933b-2d50-408a-9e4d-ce2768399c1d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.469114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193f933b-2d50-408a-9e4d-ce2768399c1d None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.469828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193f933b-2d50-408a-9e4d-ce2768399c1d None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:31.470851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193f933b-2d50-408a-9e4d-ce2768399c1d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:31.472273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193f933b-2d50-408a-9e4d-ce2768399c1d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:31.488643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 288/581] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:31.488643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193f933b-2d50-408a-9e4d-ce2768399c1d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.491566 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29bcfcd-dffd-48b8-873d-a032a6f94177 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.522386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:31.536479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d610f-a2ba-4ad1-b03d-119a460ad964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.537230 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d610f-a2ba-4ad1-b03d-119a460ad964 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.537591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d610f-a2ba-4ad1-b03d-119a460ad964 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.537941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d610f-a2ba-4ad1-b03d-119a460ad964 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.547745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:31.549829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557d610f-a2ba-4ad1-b03d-119a460ad964 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:31.551361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557d610f-a2ba-4ad1-b03d-119a460ad964 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-557d610f-a2ba-4ad1-b03d-119a460ad964 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:31.566749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:31.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:31.573754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:31.573754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:31.573754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:31.573754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:31.573754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 294/582] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:31.573754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d610f-a2ba-4ad1-b03d-119a460ad964 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.600049 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193f933b-2d50-408a-9e4d-ce2768399c1d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.635508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ca06db-dcef-44f1-8c77-a11ed14cf1ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.636139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca06db-dcef-44f1-8c77-a11ed14cf1ba None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.636517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca06db-dcef-44f1-8c77-a11ed14cf1ba None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.636885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ca06db-dcef-44f1-8c77-a11ed14cf1ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.649183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ca06db-dcef-44f1-8c77-a11ed14cf1ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:31.651369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ca06db-dcef-44f1-8c77-a11ed14cf1ba None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c5ca06db-dcef-44f1-8c77-a11ed14cf1ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:31.666537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:31.667640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:31.673747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:31.673747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:31.673747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:31.673747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:31.673747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 289/583] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:31.673747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ca06db-dcef-44f1-8c77-a11ed14cf1ba None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.680126 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d610f-a2ba-4ad1-b03d-119a460ad964 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.715392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a3ef01-e464-41bd-af32-1d0cf47b4947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.716114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a3ef01-e464-41bd-af32-1d0cf47b4947 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.716480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a3ef01-e464-41bd-af32-1d0cf47b4947 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.716853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a3ef01-e464-41bd-af32-1d0cf47b4947 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.729097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a3ef01-e464-41bd-af32-1d0cf47b4947 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:31.731224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a3ef01-e464-41bd-af32-1d0cf47b4947 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-85a3ef01-e464-41bd-af32-1d0cf47b4947 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:31.746033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:31.747114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:31.748154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:31.748154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:31.748154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:31.748154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:31.748154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 295/584] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:31.752916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a3ef01-e464-41bd-af32-1d0cf47b4947 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.781561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ca06db-dcef-44f1-8c77-a11ed14cf1ba None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.783907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:31.798100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:31.820697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c36ba9-288b-44cb-bc57-81671295ab85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.821351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c36ba9-288b-44cb-bc57-81671295ab85 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.821642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c36ba9-288b-44cb-bc57-81671295ab85 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.821918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c36ba9-288b-44cb-bc57-81671295ab85 None admin] PATH_INFO: `/v3/users/glance` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.829845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c36ba9-288b-44cb-bc57-81671295ab85 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:31.831289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c36ba9-288b-44cb-bc57-81671295ab85 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-85c36ba9-288b-44cb-bc57-81671295ab85 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:31.842755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:31.843704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 290/585] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/users/glance => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:31.844657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c36ba9-288b-44cb-bc57-81671295ab85 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.867497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a3ef01-e464-41bd-af32-1d0cf47b4947 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.903189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8ea0eb-4cf7-412a-8cf5-52c5e1e65e14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.903863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8ea0eb-4cf7-412a-8cf5-52c5e1e65e14 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.904151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8ea0eb-4cf7-412a-8cf5-52c5e1e65e14 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.904433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8ea0eb-4cf7-412a-8cf5-52c5e1e65e14 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.905186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8ea0eb-4cf7-412a-8cf5-52c5e1e65e14 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:31.905477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8ea0eb-4cf7-412a-8cf5-52c5e1e65e14 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:31.906848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8ea0eb-4cf7-412a-8cf5-52c5e1e65e14 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:31.919212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 296/586] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:31.920036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8ea0eb-4cf7-412a-8cf5-52c5e1e65e14 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:31.959416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c36ba9-288b-44cb-bc57-81671295ab85 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:31.997419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e0e160-1828-480a-80b1-715057412a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:31.998039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e0e160-1828-480a-80b1-715057412a93 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:31.998324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e0e160-1828-480a-80b1-715057412a93 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:31.998598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e0e160-1828-480a-80b1-715057412a93 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:31.999300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e0e160-1828-480a-80b1-715057412a93 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:31.999607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e0e160-1828-480a-80b1-715057412a93 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.000999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e0e160-1828-480a-80b1-715057412a93 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.013249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 291/587] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:32.014060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e0e160-1828-480a-80b1-715057412a93 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.022972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:32.036994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8ea0eb-4cf7-412a-8cf5-52c5e1e65e14 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.055875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:32.073917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac5c523-d35d-4780-b621-adad921eb467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.074603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac5c523-d35d-4780-b621-adad921eb467 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.074811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac5c523-d35d-4780-b621-adad921eb467 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.075082 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac5c523-d35d-4780-b621-adad921eb467 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.075766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac5c523-d35d-4780-b621-adad921eb467 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:32.076070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac5c523-d35d-4780-b621-adad921eb467 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.077458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac5c523-d35d-4780-b621-adad921eb467 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.090277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 297/588] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:32.090910 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac5c523-d35d-4780-b621-adad921eb467 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.132136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e0e160-1828-480a-80b1-715057412a93 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.169698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908921a2-f7da-4777-a89e-8291230baaab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.170241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908921a2-f7da-4777-a89e-8291230baaab None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.170549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908921a2-f7da-4777-a89e-8291230baaab None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.170825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908921a2-f7da-4777-a89e-8291230baaab None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.171617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908921a2-f7da-4777-a89e-8291230baaab None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:32.171919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908921a2-f7da-4777-a89e-8291230baaab None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.174343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908921a2-f7da-4777-a89e-8291230baaab None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.197034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 292/589] 10.222.0.40 () {56 vars in 1137 bytes} [Fri Jul 29 04:34:32 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:32.197034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908921a2-f7da-4777-a89e-8291230baaab None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.203108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac5c523-d35d-4780-b621-adad921eb467 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.242128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c9ed69-dba6-4871-ad70-d338cfb01af3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.243113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c9ed69-dba6-4871-ad70-d338cfb01af3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.243401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c9ed69-dba6-4871-ad70-d338cfb01af3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.243676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c9ed69-dba6-4871-ad70-d338cfb01af3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.251601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c9ed69-dba6-4871-ad70-d338cfb01af3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.253101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c9ed69-dba6-4871-ad70-d338cfb01af3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-b9c9ed69-dba6-4871-ad70-d338cfb01af3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:32.265485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:32.266488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:32.267623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:32.267623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:32.267623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:32.267623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:32.267623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:32.267623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:32.267623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:32.267623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:32.267623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 298/590] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:32.268103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c9ed69-dba6-4871-ad70-d338cfb01af3 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.279703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:32.300963 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:32.311820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908921a2-f7da-4777-a89e-8291230baaab None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.352038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e9a08-3466-4783-98d4-0b44e5bdbbbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.352814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e9a08-3466-4783-98d4-0b44e5bdbbbc None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.353106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e9a08-3466-4783-98d4-0b44e5bdbbbc None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.353389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5e9a08-3466-4783-98d4-0b44e5bdbbbc None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.360113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e9a08-3466-4783-98d4-0b44e5bdbbbc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.361791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5e9a08-3466-4783-98d4-0b44e5bdbbbc None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-ab5e9a08-3466-4783-98d4-0b44e5bdbbbc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:32.372943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:32.373814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:32.374675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:32.374675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:32.374675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:32.374675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:32.374675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:32.374675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:32.374675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:32.374675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:32.374675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 293/591] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:32.375143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c9ed69-dba6-4871-ad70-d338cfb01af3 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.375221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e9a08-3466-4783-98d4-0b44e5bdbbbc None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.409266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd767e3-a9ec-41de-8069-c2ca11e22365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.409898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd767e3-a9ec-41de-8069-c2ca11e22365 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.410180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd767e3-a9ec-41de-8069-c2ca11e22365 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.410456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd767e3-a9ec-41de-8069-c2ca11e22365 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.417675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd767e3-a9ec-41de-8069-c2ca11e22365 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.419117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd767e3-a9ec-41de-8069-c2ca11e22365 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-0cd767e3-a9ec-41de-8069-c2ca11e22365 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:32.430742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:32.431747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 299/592] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:32 2022] GET /identity/v3/users/neutron => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:32.433250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd767e3-a9ec-41de-8069-c2ca11e22365 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.480293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5e9a08-3466-4783-98d4-0b44e5bdbbbc None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.499033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:32.513925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86f2ee5-e1db-473d-8108-5d3f059887db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.514566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86f2ee5-e1db-473d-8108-5d3f059887db None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.514849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86f2ee5-e1db-473d-8108-5d3f059887db None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.515129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86f2ee5-e1db-473d-8108-5d3f059887db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.525537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86f2ee5-e1db-473d-8108-5d3f059887db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.526981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86f2ee5-e1db-473d-8108-5d3f059887db None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.533410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd767e3-a9ec-41de-8069-c2ca11e22365 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c86f2ee5-e1db-473d-8108-5d3f059887db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:32.541660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:32.542556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:32.543411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:32.543411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:32.543411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:32.543411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:32.543411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 294/593] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:32.544037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42699c49-7fe8-4d06-a5d8-22e792b11813 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.544297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42699c49-7fe8-4d06-a5d8-22e792b11813 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.544578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42699c49-7fe8-4d06-a5d8-22e792b11813 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.545837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 295/594] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:32.546532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42699c49-7fe8-4d06-a5d8-22e792b11813 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.551387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:32.571868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fcf688-4419-476c-a64a-da7ddede0664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.572526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fcf688-4419-476c-a64a-da7ddede0664 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.572798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fcf688-4419-476c-a64a-da7ddede0664 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.573073 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fcf688-4419-476c-a64a-da7ddede0664 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.573821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fcf688-4419-476c-a64a-da7ddede0664 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:32.574135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fcf688-4419-476c-a64a-da7ddede0664 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.575548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fcf688-4419-476c-a64a-da7ddede0664 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.587711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 300/595] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:32.588494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fcf688-4419-476c-a64a-da7ddede0664 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.654055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42699c49-7fe8-4d06-a5d8-22e792b11813 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.688166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0169b2c8-6243-48fa-ab86-8d6235f7017c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.689271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0169b2c8-6243-48fa-ab86-8d6235f7017c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.689560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0169b2c8-6243-48fa-ab86-8d6235f7017c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.689975 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0169b2c8-6243-48fa-ab86-8d6235f7017c None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.690727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0169b2c8-6243-48fa-ab86-8d6235f7017c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:32.691038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0169b2c8-6243-48fa-ab86-8d6235f7017c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.692465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0169b2c8-6243-48fa-ab86-8d6235f7017c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.693008 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fcf688-4419-476c-a64a-da7ddede0664 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.704050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 296/596] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:32.705208 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0169b2c8-6243-48fa-ab86-8d6235f7017c None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.720351 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:32.728078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4e8532-775b-4028-9e5d-1db3c97cd37e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.728731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4e8532-775b-4028-9e5d-1db3c97cd37e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.729012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4e8532-775b-4028-9e5d-1db3c97cd37e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.729287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4e8532-775b-4028-9e5d-1db3c97cd37e None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.730079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4e8532-775b-4028-9e5d-1db3c97cd37e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:32.730385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4e8532-775b-4028-9e5d-1db3c97cd37e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.731795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4e8532-775b-4028-9e5d-1db3c97cd37e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.749323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 301/597] 10.222.0.40 () {56 vars in 1139 bytes} [Fri Jul 29 04:34:32 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:32.750533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d864e27-ff01-4148-923b-992bcf1215f6 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.750819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d864e27-ff01-4148-923b-992bcf1215f6 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.751740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d864e27-ff01-4148-923b-992bcf1215f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.768804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2d864e27-ff01-4148-923b-992bcf1215f6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:32.770006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2d864e27-ff01-4148-923b-992bcf1215f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:32.774269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:32.812304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0169b2c8-6243-48fa-ab86-8d6235f7017c None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.836644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 302/598] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:32.837457 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d864e27-ff01-4148-923b-992bcf1215f6 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.863914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656df4f8-571b-444d-9bab-e667ca6c4148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.864537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656df4f8-571b-444d-9bab-e667ca6c4148 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.864850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656df4f8-571b-444d-9bab-e667ca6c4148 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.865127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656df4f8-571b-444d-9bab-e667ca6c4148 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.865830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656df4f8-571b-444d-9bab-e667ca6c4148 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:32.866140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656df4f8-571b-444d-9bab-e667ca6c4148 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.867505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-656df4f8-571b-444d-9bab-e667ca6c4148 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:32.879323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 297/599] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:32.880095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656df4f8-571b-444d-9bab-e667ca6c4148 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:32.937271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d864e27-ff01-4148-923b-992bcf1215f6 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.953880 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:32.972427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c56b02-bdb3-48bb-bbd2-35ba2a270124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:32.973143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c56b02-bdb3-48bb-bbd2-35ba2a270124 None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:32.973424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c56b02-bdb3-48bb-bbd2-35ba2a270124 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:32.973712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c56b02-bdb3-48bb-bbd2-35ba2a270124 None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/58c33a72a871432b88da4f0bf1bc514c/roles/7140c1d0b4f74727a1885faa13c9e6ec` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:32.981020 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656df4f8-571b-444d-9bab-e667ca6c4148 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:32.989758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c56b02-bdb3-48bb-bbd2-35ba2a270124 None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=58c33a72a871432b88da4f0bf1bc514c, role_id=7140c1d0b4f74727a1885faa13c9e6ec)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:32.991221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c56b02-bdb3-48bb-bbd2-35ba2a270124 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:33.004047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 303/600] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:34:32 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/58c33a72a871432b88da4f0bf1bc514c/roles/7140c1d0b4f74727a1885faa13c9e6ec => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:33.004867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c56b02-bdb3-48bb-bbd2-35ba2a270124 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:33.010025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:33.023050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0a21e5-0030-4b4b-a451-66501a1f648d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:33.023866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a21e5-0030-4b4b-a451-66501a1f648d None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:33.024168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a21e5-0030-4b4b-a451-66501a1f648d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:33.024434 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0a21e5-0030-4b4b-a451-66501a1f648d None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/de49172e2a054096b319a7e4349ecaea/roles/7140c1d0b4f74727a1885faa13c9e6ec` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:33.041480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0a21e5-0030-4b4b-a451-66501a1f648d None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=de49172e2a054096b319a7e4349ecaea, role_id=7140c1d0b4f74727a1885faa13c9e6ec)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:33.043059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0a21e5-0030-4b4b-a451-66501a1f648d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:33.057643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 298/601] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:34:32 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/de49172e2a054096b319a7e4349ecaea/roles/7140c1d0b4f74727a1885faa13c9e6ec => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:33.058259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0a21e5-0030-4b4b-a451-66501a1f648d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:33.162638 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c56b02-bdb3-48bb-bbd2-35ba2a270124 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:33.197623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0a21e5-0030-4b4b-a451-66501a1f648d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:33.215930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98d27b2-e344-4d81-bfc1-48f1d2849b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:33.216510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98d27b2-e344-4d81-bfc1-48f1d2849b7c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:33.216819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98d27b2-e344-4d81-bfc1-48f1d2849b7c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:33.217099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98d27b2-e344-4d81-bfc1-48f1d2849b7c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:33.229836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98d27b2-e344-4d81-bfc1-48f1d2849b7c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:33.231646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98d27b2-e344-4d81-bfc1-48f1d2849b7c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:33.243870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce624111-e3dd-4036-9d01-f121fc08a045 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:33.244517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce624111-e3dd-4036-9d01-f121fc08a045 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:33.244856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce624111-e3dd-4036-9d01-f121fc08a045 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:33.245133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce624111-e3dd-4036-9d01-f121fc08a045 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-d98d27b2-e344-4d81-bfc1-48f1d2849b7c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:33.248637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:33.249783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:33.250663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:33.250663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:33.250663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:33.250663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:33.256017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 304/602] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:33.256017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98d27b2-e344-4d81-bfc1-48f1d2849b7c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:33.260645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce624111-e3dd-4036-9d01-f121fc08a045 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-ce624111-e3dd-4036-9d01-f121fc08a045 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:33.264382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:33.265486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:33.266294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:33.273390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 299/603] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:33.273390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce624111-e3dd-4036-9d01-f121fc08a045 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:33.287133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:33.309412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:33.388098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce624111-e3dd-4036-9d01-f121fc08a045 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:33.407786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98d27b2-e344-4d81-bfc1-48f1d2849b7c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:33.427123 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629725e5-a088-4707-b5c9-d01c5cc1347a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:33.427752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629725e5-a088-4707-b5c9-d01c5cc1347a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:33.428034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629725e5-a088-4707-b5c9-d01c5cc1347a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:33.428310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629725e5-a088-4707-b5c9-d01c5cc1347a None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:33.429040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629725e5-a088-4707-b5c9-d01c5cc1347a None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:33.429344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629725e5-a088-4707-b5c9-d01c5cc1347a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:33.430844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629725e5-a088-4707-b5c9-d01c5cc1347a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:33.447230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 300/604] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:33.447230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629725e5-a088-4707-b5c9-d01c5cc1347a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:33.462657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6920c84e-11b6-4d70-baf5-38486789c712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:33.463566 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6920c84e-11b6-4d70-baf5-38486789c712 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:33.464265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6920c84e-11b6-4d70-baf5-38486789c712 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:33.464727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6920c84e-11b6-4d70-baf5-38486789c712 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:33.474980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6920c84e-11b6-4d70-baf5-38486789c712 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:33.477418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6920c84e-11b6-4d70-baf5-38486789c712 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-6920c84e-11b6-4d70-baf5-38486789c712 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:33.494196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:33.495334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:33.496479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:33.496479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:33.496479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:33.496479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:33.496479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:33.496479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:33.496479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:33.496479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:33.499356 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 305/605] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:33.500681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6920c84e-11b6-4d70-baf5-38486789c712 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:33.553384 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:33.560188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629725e5-a088-4707-b5c9-d01c5cc1347a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:33.603848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0344d34d-f059-4103-a4ce-79e82831c52b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:33.604905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0344d34d-f059-4103-a4ce-79e82831c52b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:33.605305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0344d34d-f059-4103-a4ce-79e82831c52b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:33.606548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0344d34d-f059-4103-a4ce-79e82831c52b None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:33.607379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0344d34d-f059-4103-a4ce-79e82831c52b None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:33.607840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0344d34d-f059-4103-a4ce-79e82831c52b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:33.610655 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0344d34d-f059-4103-a4ce-79e82831c52b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:33.618179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:33.624634 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 301/606] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:33.625856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0344d34d-f059-4103-a4ce-79e82831c52b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:33.646856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6920c84e-11b6-4d70-baf5-38486789c712 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:33.693443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8864066-1925-4459-99f6-3ff2270e39c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:33.694362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8864066-1925-4459-99f6-3ff2270e39c0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:33.695043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8864066-1925-4459-99f6-3ff2270e39c0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:33.695466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8864066-1925-4459-99f6-3ff2270e39c0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:33.705362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8864066-1925-4459-99f6-3ff2270e39c0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:33.707778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8864066-1925-4459-99f6-3ff2270e39c0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-d8864066-1925-4459-99f6-3ff2270e39c0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:33.723767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:33.724891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:33.726031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:33.726031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:33.726031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:33.726031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:33.726031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:33.726031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:33.726031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:33.726031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:33.733554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 306/607] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:33.733554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8864066-1925-4459-99f6-3ff2270e39c0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:33.765661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0344d34d-f059-4103-a4ce-79e82831c52b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:33.812699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e309545-7477-40c4-9af7-53b030def269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:33.813980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e309545-7477-40c4-9af7-53b030def269 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:33.814717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e309545-7477-40c4-9af7-53b030def269 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:33.815237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e309545-7477-40c4-9af7-53b030def269 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:33.816337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e309545-7477-40c4-9af7-53b030def269 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:33.816799 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e309545-7477-40c4-9af7-53b030def269 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:33.819079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e309545-7477-40c4-9af7-53b030def269 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:33.836038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 302/608] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:33.837064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e309545-7477-40c4-9af7-53b030def269 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:33.862493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:33.870405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8864066-1925-4459-99f6-3ff2270e39c0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:33.917427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ec3ee6-106f-435f-b9ad-ac4ac3d190e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:33.918303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec3ee6-106f-435f-b9ad-ac4ac3d190e3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:33.918729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec3ee6-106f-435f-b9ad-ac4ac3d190e3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:33.919405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec3ee6-106f-435f-b9ad-ac4ac3d190e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:33.929053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:33.935746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ec3ee6-106f-435f-b9ad-ac4ac3d190e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:33.937685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ec3ee6-106f-435f-b9ad-ac4ac3d190e3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:33.957581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e309545-7477-40c4-9af7-53b030def269 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-10ec3ee6-106f-435f-b9ad-ac4ac3d190e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:33.957970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:33.959009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:33.960081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:33.960081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:33.960081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:33.960081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:33.962988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 307/609] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:33.963985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ec3ee6-106f-435f-b9ad-ac4ac3d190e3 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:33.998096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f0128d-bade-476b-9cf6-70cfcab3468d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:33.999091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f0128d-bade-476b-9cf6-70cfcab3468d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:33.999482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f0128d-bade-476b-9cf6-70cfcab3468d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:33.999905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f0128d-bade-476b-9cf6-70cfcab3468d None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:34.000808 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f0128d-bade-476b-9cf6-70cfcab3468d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:34.001606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f0128d-bade-476b-9cf6-70cfcab3468d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:34.003052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f0128d-bade-476b-9cf6-70cfcab3468d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:34.022237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 303/610] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:34.022237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f0128d-bade-476b-9cf6-70cfcab3468d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:34.102584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ec3ee6-106f-435f-b9ad-ac4ac3d190e3 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:34.108399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:34.140861 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f0128d-bade-476b-9cf6-70cfcab3468d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:34.145370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aaa822-6390-4fb3-815f-03cec3708b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:34.146296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aaa822-6390-4fb3-815f-03cec3708b5d None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:34.146725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aaa822-6390-4fb3-815f-03cec3708b5d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:34.147182 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aaa822-6390-4fb3-815f-03cec3708b5d None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/3749dc021aac478f88072ced48f4cee4/roles/7140c1d0b4f74727a1885faa13c9e6ec` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:34.168150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27aaa822-6390-4fb3-815f-03cec3708b5d None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=3749dc021aac478f88072ced48f4cee4, role_id=7140c1d0b4f74727a1885faa13c9e6ec)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:34.170839 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27aaa822-6390-4fb3-815f-03cec3708b5d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:34.180234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e3bea6-236e-4dc6-803f-f223cd386264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:34.180234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e3bea6-236e-4dc6-803f-f223cd386264 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:34.180481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e3bea6-236e-4dc6-803f-f223cd386264 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:34.180909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e3bea6-236e-4dc6-803f-f223cd386264 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:34.181681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e3bea6-236e-4dc6-803f-f223cd386264 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:34.182060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e3bea6-236e-4dc6-803f-f223cd386264 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:34.183522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e3bea6-236e-4dc6-803f-f223cd386264 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:34.200634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 308/611] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:34:33 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/3749dc021aac478f88072ced48f4cee4/roles/7140c1d0b4f74727a1885faa13c9e6ec => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:34.200634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aaa822-6390-4fb3-815f-03cec3708b5d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:34.210317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 304/612] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:34.215848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e3bea6-236e-4dc6-803f-f223cd386264 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:34.290295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:34.408566 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aaa822-6390-4fb3-815f-03cec3708b5d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:34.419412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e3bea6-236e-4dc6-803f-f223cd386264 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:34.460424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b932a56-bf06-4c54-96ff-15b4b0d130da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:34.461153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0607b374-1977-48ba-9652-29308bd15941 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:34.461770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b932a56-bf06-4c54-96ff-15b4b0d130da None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:34.461882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0607b374-1977-48ba-9652-29308bd15941 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:34.462060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0607b374-1977-48ba-9652-29308bd15941 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:34.462338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0607b374-1977-48ba-9652-29308bd15941 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:34.462757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b932a56-bf06-4c54-96ff-15b4b0d130da None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:34.463237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b932a56-bf06-4c54-96ff-15b4b0d130da None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/f6726ef0c4184040a18a76898bb738d1/roles/7140c1d0b4f74727a1885faa13c9e6ec` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:34.465922 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:34.472097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0607b374-1977-48ba-9652-29308bd15941 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:34.474140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0607b374-1977-48ba-9652-29308bd15941 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:34.486063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b932a56-bf06-4c54-96ff-15b4b0d130da None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=f6726ef0c4184040a18a76898bb738d1, role_id=7140c1d0b4f74727a1885faa13c9e6ec)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:34.487721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b932a56-bf06-4c54-96ff-15b4b0d130da None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-0607b374-1977-48ba-9652-29308bd15941 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:34.489890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:34.491044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:34.492178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:34.492178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:34.492178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:34.492178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:34.492178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:34.492178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:34.492178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:34.492178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:34.492178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:34.492178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:34.494039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 305/613] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:34 2022] GET /identity/v3/users/swift => generated 83 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:34.499529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0607b374-1977-48ba-9652-29308bd15941 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:34.509535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 309/614] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:34:34 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/f6726ef0c4184040a18a76898bb738d1/roles/7140c1d0b4f74727a1885faa13c9e6ec => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:34.621752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0607b374-1977-48ba-9652-29308bd15941 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:34.662601 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8a73b6-aade-4a04-8e1c-b1ce9f4cd34c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:34.663534 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8a73b6-aade-4a04-8e1c-b1ce9f4cd34c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:34.663971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8a73b6-aade-4a04-8e1c-b1ce9f4cd34c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:34.664398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e8a73b6-aade-4a04-8e1c-b1ce9f4cd34c None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:34.665403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8a73b6-aade-4a04-8e1c-b1ce9f4cd34c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:34.665889 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8a73b6-aade-4a04-8e1c-b1ce9f4cd34c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:34.667537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e8a73b6-aade-4a04-8e1c-b1ce9f4cd34c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:34.689486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 306/615] 10.222.0.40 () {56 vars in 1135 bytes} [Fri Jul 29 04:34:34 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:34.694656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b932a56-bf06-4c54-96ff-15b4b0d130da None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:34.796096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:34.806450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b932a56-bf06-4c54-96ff-15b4b0d130da None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:34.844407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7c4e5c-78c8-49e9-a5ea-fe49faa04512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:34.845393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7c4e5c-78c8-49e9-a5ea-fe49faa04512 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:34.845914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7c4e5c-78c8-49e9-a5ea-fe49faa04512 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:34.846295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7c4e5c-78c8-49e9-a5ea-fe49faa04512 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:34.862220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7c4e5c-78c8-49e9-a5ea-fe49faa04512 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:34.863837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7c4e5c-78c8-49e9-a5ea-fe49faa04512 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-7b7c4e5c-78c8-49e9-a5ea-fe49faa04512 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:34.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:34.884468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:34.885417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:34.885417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:34.885417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:34.885417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:34.888130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 310/616] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:34.890141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8a73b6-aade-4a04-8e1c-b1ce9f4cd34c None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:34.988133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:35.045860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e8a73b6-aade-4a04-8e1c-b1ce9f4cd34c None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:35.099150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1cba33-3908-481e-8c1c-0d5b8f2528cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:35.100002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1cba33-3908-481e-8c1c-0d5b8f2528cc None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.100392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1cba33-3908-481e-8c1c-0d5b8f2528cc None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.100847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1cba33-3908-481e-8c1c-0d5b8f2528cc None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.101903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1cba33-3908-481e-8c1c-0d5b8f2528cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:35.102298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1cba33-3908-481e-8c1c-0d5b8f2528cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:35.103700 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1cba33-3908-481e-8c1c-0d5b8f2528cc None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:35.124013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 307/617] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:35.129781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7c4e5c-78c8-49e9-a5ea-fe49faa04512 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:35.266278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7c4e5c-78c8-49e9-a5ea-fe49faa04512 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:35.314959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0b2294-2f07-4816-b28a-1de5848b45a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:35.315587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0b2294-2f07-4816-b28a-1de5848b45a8 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.315876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0b2294-2f07-4816-b28a-1de5848b45a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.316153 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0b2294-2f07-4816-b28a-1de5848b45a8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.323367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f379f6a-f298-4140-bfc2-cd6a52548ad0 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.323660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f379f6a-f298-4140-bfc2-cd6a52548ad0 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.323943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f379f6a-f298-4140-bfc2-cd6a52548ad0 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.325839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 308/618] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:35.326127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0b2294-2f07-4816-b28a-1de5848b45a8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:35.327586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0b2294-2f07-4816-b28a-1de5848b45a8 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:35.335452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34265654-4f88-4f29-9ea0-91aca5a651ce None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.335745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34265654-4f88-4f29-9ea0-91aca5a651ce None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.336033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34265654-4f88-4f29-9ea0-91aca5a651ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-7b0b2294-2f07-4816-b28a-1de5848b45a8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:35.343999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:35.344916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:35.345942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:35.345942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:35.345942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:35.345942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:35.345942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:35.345942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:35.345942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:35.345942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:35.345942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 311/619] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:35.351051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0b2294-2f07-4816-b28a-1de5848b45a8 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:35.353747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:35.374801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-34265654-4f88-4f29-9ea0-91aca5a651ce None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:35.377466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-34265654-4f88-4f29-9ea0-91aca5a651ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:35.453928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 309/620] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:35.455373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d584ff8-3d5d-4476-a534-cc9f765eded7 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.455661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d584ff8-3d5d-4476-a534-cc9f765eded7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.455939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d584ff8-3d5d-4476-a534-cc9f765eded7 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.461471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 310/621] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:35.463111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199ec9e-1967-4662-a80a-9a257ed43caa None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.463406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199ec9e-1967-4662-a80a-9a257ed43caa None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.463784 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199ec9e-1967-4662-a80a-9a257ed43caa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.473116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0b2294-2f07-4816-b28a-1de5848b45a8 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:35.483790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-3199ec9e-1967-4662-a80a-9a257ed43caa None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:35.485115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3199ec9e-1967-4662-a80a-9a257ed43caa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:35.509613 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103cfe71-72ed-4534-b355-a3b1fc3ad81b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:35.510246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103cfe71-72ed-4534-b355-a3b1fc3ad81b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.510540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103cfe71-72ed-4534-b355-a3b1fc3ad81b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.510824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103cfe71-72ed-4534-b355-a3b1fc3ad81b None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.511651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103cfe71-72ed-4534-b355-a3b1fc3ad81b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:35.511973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103cfe71-72ed-4534-b355-a3b1fc3ad81b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:35.513494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103cfe71-72ed-4534-b355-a3b1fc3ad81b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:35.527649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 312/622] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:35.528488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103cfe71-72ed-4534-b355-a3b1fc3ad81b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:35.528955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:35.564726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 311/623] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:35.565755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3199ec9e-1967-4662-a80a-9a257ed43caa None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:35.596033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:35.640793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103cfe71-72ed-4534-b355-a3b1fc3ad81b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:35.685628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3199ec9e-1967-4662-a80a-9a257ed43caa None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:35.693151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d008ca4-706c-48fe-9463-31880b21bcf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:35.694035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d008ca4-706c-48fe-9463-31880b21bcf5 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.694472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d008ca4-706c-48fe-9463-31880b21bcf5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.694902 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d008ca4-706c-48fe-9463-31880b21bcf5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.706096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d008ca4-706c-48fe-9463-31880b21bcf5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-3d008ca4-706c-48fe-9463-31880b21bcf5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:35.711388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:35.712476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:35.713542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:35.715836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 313/624] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:35.716789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d008ca4-706c-48fe-9463-31880b21bcf5 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:35.724465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b146151-547b-46b7-b933-2ff8fa0fa0c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:35.725309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b146151-547b-46b7-b933-2ff8fa0fa0c4 None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.725693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b146151-547b-46b7-b933-2ff8fa0fa0c4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.726076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b146151-547b-46b7-b933-2ff8fa0fa0c4 None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/bbfe92babca048d882cb59cb29b3ed97/roles/7140c1d0b4f74727a1885faa13c9e6ec` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.741291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b146151-547b-46b7-b933-2ff8fa0fa0c4 None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=bbfe92babca048d882cb59cb29b3ed97, role_id=7140c1d0b4f74727a1885faa13c9e6ec)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:35.742894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b146151-547b-46b7-b933-2ff8fa0fa0c4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:35.758800 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 312/625] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:34:35 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/bbfe92babca048d882cb59cb29b3ed97/roles/7140c1d0b4f74727a1885faa13c9e6ec => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:35.759063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b146151-547b-46b7-b933-2ff8fa0fa0c4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:35.786849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:35.834348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d008ca4-706c-48fe-9463-31880b21bcf5 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:35.861220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:35.876878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b146151-547b-46b7-b933-2ff8fa0fa0c4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:35.877445 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b418b855-f4a5-419a-806f-f2edae609320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:35.878269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418b855-f4a5-419a-806f-f2edae609320 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.878690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418b855-f4a5-419a-806f-f2edae609320 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.879105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b418b855-f4a5-419a-806f-f2edae609320 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.911309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b418b855-f4a5-419a-806f-f2edae609320 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-b418b855-f4a5-419a-806f-f2edae609320 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:35.924360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:35.925621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:35.926533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:35.928523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 314/626] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:35.935283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b418b855-f4a5-419a-806f-f2edae609320 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:35.971032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b73d10-b367-4fea-9071-7d6e52068e7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:35.971960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b73d10-b367-4fea-9071-7d6e52068e7d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:35.972797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b73d10-b367-4fea-9071-7d6e52068e7d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:35.974578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b73d10-b367-4fea-9071-7d6e52068e7d None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:35.975497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b73d10-b367-4fea-9071-7d6e52068e7d None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:35.976371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b73d10-b367-4fea-9071-7d6e52068e7d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:35.981071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b73d10-b367-4fea-9071-7d6e52068e7d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:35.996557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 313/627] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:36.002933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b73d10-b367-4fea-9071-7d6e52068e7d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:36.095924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b418b855-f4a5-419a-806f-f2edae609320 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:36.120309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:36.135388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b73d10-b367-4fea-9071-7d6e52068e7d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:36.140532 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcef6054-4dd6-407c-9377-1b9151ea8e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:36.141442 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcef6054-4dd6-407c-9377-1b9151ea8e10 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:36.141985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcef6054-4dd6-407c-9377-1b9151ea8e10 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:36.142470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcef6054-4dd6-407c-9377-1b9151ea8e10 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:36.143391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcef6054-4dd6-407c-9377-1b9151ea8e10 None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:36.143856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcef6054-4dd6-407c-9377-1b9151ea8e10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:36.146058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcef6054-4dd6-407c-9377-1b9151ea8e10 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:36.158807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 315/628] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:36.164448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcef6054-4dd6-407c-9377-1b9151ea8e10 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:36.178604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c993264-85a1-47b9-bd3f-168a04c70ddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:36.179300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c993264-85a1-47b9-bd3f-168a04c70ddd None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:36.179592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c993264-85a1-47b9-bd3f-168a04c70ddd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:36.179875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c993264-85a1-47b9-bd3f-168a04c70ddd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:36.192520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c993264-85a1-47b9-bd3f-168a04c70ddd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:36.194390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c993264-85a1-47b9-bd3f-168a04c70ddd None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-5c993264-85a1-47b9-bd3f-168a04c70ddd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:36.210742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:36.211972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:36.213098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:36.213098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:36.213098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:36.213098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:36.214952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 314/629] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:36.221361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c993264-85a1-47b9-bd3f-168a04c70ddd None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:36.328960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:36.339368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c993264-85a1-47b9-bd3f-168a04c70ddd None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:36.396406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7827c77-7e31-414d-9b95-8a18736c056d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:36.397470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7827c77-7e31-414d-9b95-8a18736c056d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:36.400439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7827c77-7e31-414d-9b95-8a18736c056d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:36.400892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7827c77-7e31-414d-9b95-8a18736c056d None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:36.401739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7827c77-7e31-414d-9b95-8a18736c056d None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:36.402444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7827c77-7e31-414d-9b95-8a18736c056d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:36.403974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7827c77-7e31-414d-9b95-8a18736c056d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:36.418471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:36.426474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcef6054-4dd6-407c-9377-1b9151ea8e10 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:36.430236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 315/630] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:36.438634 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7827c77-7e31-414d-9b95-8a18736c056d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:36.488217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8698325d-48cc-4168-936c-2c2557f59a8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:36.488967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8698325d-48cc-4168-936c-2c2557f59a8e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:36.489253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8698325d-48cc-4168-936c-2c2557f59a8e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:36.489533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8698325d-48cc-4168-936c-2c2557f59a8e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:36.503608 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8698325d-48cc-4168-936c-2c2557f59a8e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:36.505099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8698325d-48cc-4168-936c-2c2557f59a8e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-8698325d-48cc-4168-936c-2c2557f59a8e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:36.530318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:36.531239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:36.532112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:36.532112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:36.532112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:36.532112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:36.533762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 316/631] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 368 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:36.536974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8698325d-48cc-4168-936c-2c2557f59a8e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:36.632118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7827c77-7e31-414d-9b95-8a18736c056d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:36.709797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:36.717466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8698325d-48cc-4168-936c-2c2557f59a8e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:36.725087 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f407b9a-18ec-402f-9826-bf8433eb713e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:36.725761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f407b9a-18ec-402f-9826-bf8433eb713e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:36.726059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f407b9a-18ec-402f-9826-bf8433eb713e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:36.726340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f407b9a-18ec-402f-9826-bf8433eb713e None admin] PATH_INFO: `/v3/users/cinder` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:36.738277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f407b9a-18ec-402f-9826-bf8433eb713e None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:36.739915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f407b9a-18ec-402f-9826-bf8433eb713e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-8f407b9a-18ec-402f-9826-bf8433eb713e None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:36.756195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:36.757271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:36.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 316/632] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:36 2022] GET /identity/v3/users/cinder => generated 84 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:36.760419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eda850-04b5-43b4-a705-fb8f60ed1bac None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:36.760419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eda850-04b5-43b4-a705-fb8f60ed1bac None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:36.760419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04eda850-04b5-43b4-a705-fb8f60ed1bac None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:36.764624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 317/633] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:36.764624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04eda850-04b5-43b4-a705-fb8f60ed1bac None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:36.770347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe23f32-bb8f-4da0-a9df-e17613338d35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:36.770977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe23f32-bb8f-4da0-a9df-e17613338d35 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:36.771257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe23f32-bb8f-4da0-a9df-e17613338d35 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:36.771530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe23f32-bb8f-4da0-a9df-e17613338d35 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:36.772226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe23f32-bb8f-4da0-a9df-e17613338d35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:36.772533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe23f32-bb8f-4da0-a9df-e17613338d35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:36.773925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe23f32-bb8f-4da0-a9df-e17613338d35 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:36.791176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 317/634] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:36.793485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d6faac-889e-4823-9fc1-f52b86b55e3b None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:36.794001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d6faac-889e-4823-9fc1-f52b86b55e3b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:36.794499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d6faac-889e-4823-9fc1-f52b86b55e3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:36.817272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-30d6faac-889e-4823-9fc1-f52b86b55e3b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:36.818473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-30d6faac-889e-4823-9fc1-f52b86b55e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:36.841427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:36.892687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 318/635] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:36.892687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d6faac-889e-4823-9fc1-f52b86b55e3b None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:36.894982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04eda850-04b5-43b4-a705-fb8f60ed1bac None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:36.950673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cef3246-d960-4160-8d70-76c940b04ff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:36.951647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cef3246-d960-4160-8d70-76c940b04ff3 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:36.952242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cef3246-d960-4160-8d70-76c940b04ff3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:36.952890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cef3246-d960-4160-8d70-76c940b04ff3 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:36.953908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cef3246-d960-4160-8d70-76c940b04ff3 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:36.955176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cef3246-d960-4160-8d70-76c940b04ff3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:36.957104 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cef3246-d960-4160-8d70-76c940b04ff3 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:36.980577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 318/636] 10.222.0.40 () {56 vars in 1137 bytes} [Fri Jul 29 04:34:36 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:36.981819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cef3246-d960-4160-8d70-76c940b04ff3 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:36.982760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:36.998054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d6faac-889e-4823-9fc1-f52b86b55e3b None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:37.034326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd937c5-716b-40a8-9846-20506c9f914d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:37.034974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd937c5-716b-40a8-9846-20506c9f914d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.035260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd937c5-716b-40a8-9846-20506c9f914d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.035541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd937c5-716b-40a8-9846-20506c9f914d None admin] PATH_INFO: `/v3/users/nova` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.043116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd937c5-716b-40a8-9846-20506c9f914d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:37.044580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd937c5-716b-40a8-9846-20506c9f914d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-cbd937c5-716b-40a8-9846-20506c9f914d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:37.058587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:37.059583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:37.060491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 319/637] 10.222.0.40 () {56 vars in 1093 bytes} [Fri Jul 29 04:34:36 2022] GET /identity/v3/users/nova => generated 82 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:37.061747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd937c5-716b-40a8-9846-20506c9f914d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:37.097718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cef3246-d960-4160-8d70-76c940b04ff3 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:37.124071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:37.143853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8164aa11-270f-4e99-9d4d-e43de05106ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:37.144510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8164aa11-270f-4e99-9d4d-e43de05106ad None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.144845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8164aa11-270f-4e99-9d4d-e43de05106ad None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.145259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8164aa11-270f-4e99-9d4d-e43de05106ad None admin] PATH_INFO: `/v3/roles/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.155894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8164aa11-270f-4e99-9d4d-e43de05106ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-8164aa11-270f-4e99-9d4d-e43de05106ad None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:37.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:37.161955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:37.162780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 319/638] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:37.163486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8164aa11-270f-4e99-9d4d-e43de05106ad None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:37.178107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd937c5-716b-40a8-9846-20506c9f914d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:37.214245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4077fdaa-a90d-42b3-8126-a1d18efabc8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:37.215094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4077fdaa-a90d-42b3-8126-a1d18efabc8d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.215549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4077fdaa-a90d-42b3-8126-a1d18efabc8d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.215973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4077fdaa-a90d-42b3-8126-a1d18efabc8d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.233159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:37.269872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8164aa11-270f-4e99-9d4d-e43de05106ad None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:37.290342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4077fdaa-a90d-42b3-8126-a1d18efabc8d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:37.291813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4077fdaa-a90d-42b3-8126-a1d18efabc8d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:37.305661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173f4c4e-7401-413e-bebc-d7fb49eec506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:37.306585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173f4c4e-7401-413e-bebc-d7fb49eec506 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.307107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173f4c4e-7401-413e-bebc-d7fb49eec506 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.307575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173f4c4e-7401-413e-bebc-d7fb49eec506 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.308556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173f4c4e-7401-413e-bebc-d7fb49eec506 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:37.309196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173f4c4e-7401-413e-bebc-d7fb49eec506 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:37.310917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173f4c4e-7401-413e-bebc-d7fb49eec506 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:37.330771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 320/639] 10.222.0.40 () {56 vars in 1133 bytes} [Fri Jul 29 04:34:37 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:37.332353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a51ae-f94b-4f4c-af6d-5dbc71338578 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.332664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a51ae-f94b-4f4c-af6d-5dbc71338578 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.332948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64a51ae-f94b-4f4c-af6d-5dbc71338578 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.336646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 321/640] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:37.336646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a51ae-f94b-4f4c-af6d-5dbc71338578 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:37.376821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:37.401645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 320/641] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:37.402477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4077fdaa-a90d-42b3-8126-a1d18efabc8d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:37.455279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64a51ae-f94b-4f4c-af6d-5dbc71338578 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:37.459198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:37.491829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4edbd8-64d4-4ace-9f40-941beab06de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:37.492463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4edbd8-64d4-4ace-9f40-941beab06de1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.492779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4edbd8-64d4-4ace-9f40-941beab06de1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.493061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4edbd8-64d4-4ace-9f40-941beab06de1 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.493785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4edbd8-64d4-4ace-9f40-941beab06de1 None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:37.494093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4edbd8-64d4-4ace-9f40-941beab06de1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:37.495468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4edbd8-64d4-4ace-9f40-941beab06de1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:37.504250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 322/642] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:37.505503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76b45f2-e51b-467b-8e1b-a0f5b654f640 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.505797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76b45f2-e51b-467b-8e1b-a0f5b654f640 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.506081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76b45f2-e51b-467b-8e1b-a0f5b654f640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.524305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e76b45f2-e51b-467b-8e1b-a0f5b654f640 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:37.525539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e76b45f2-e51b-467b-8e1b-a0f5b654f640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:37.532874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4077fdaa-a90d-42b3-8126-a1d18efabc8d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:37.572957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db107e0f-42c3-4450-be86-7d5ff5d887fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:37.573583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db107e0f-42c3-4450-be86-7d5ff5d887fe None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.573871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db107e0f-42c3-4450-be86-7d5ff5d887fe None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.574151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db107e0f-42c3-4450-be86-7d5ff5d887fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.604644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 323/643] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:37.604644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76b45f2-e51b-467b-8e1b-a0f5b654f640 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:37.638395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:37.652994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db107e0f-42c3-4450-be86-7d5ff5d887fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:37.654722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db107e0f-42c3-4450-be86-7d5ff5d887fe None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:37.704291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:37.723033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76b45f2-e51b-467b-8e1b-a0f5b654f640 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:37.760553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70d3152-e27b-4c40-b520-82111a38d5f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:37.761197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70d3152-e27b-4c40-b520-82111a38d5f4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.761458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70d3152-e27b-4c40-b520-82111a38d5f4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.761705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70d3152-e27b-4c40-b520-82111a38d5f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.762003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 321/644] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:37.762911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db107e0f-42c3-4450-be86-7d5ff5d887fe None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:37.777855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70d3152-e27b-4c40-b520-82111a38d5f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:37.779478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70d3152-e27b-4c40-b520-82111a38d5f4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-f70d3152-e27b-4c40-b520-82111a38d5f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:37.795136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:37.796113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:37.797074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:37.797074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:37.797074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:37.797074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:37.797074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 324/645] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:37.797074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70d3152-e27b-4c40-b520-82111a38d5f4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:37.880937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:37.882454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db107e0f-42c3-4450-be86-7d5ff5d887fe None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:37.904204 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70d3152-e27b-4c40-b520-82111a38d5f4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:37.919294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176aa27b-2881-4c6b-b17c-fa18a33175de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:37.919931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176aa27b-2881-4c6b-b17c-fa18a33175de None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.920221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176aa27b-2881-4c6b-b17c-fa18a33175de None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.920497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176aa27b-2881-4c6b-b17c-fa18a33175de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.933664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176aa27b-2881-4c6b-b17c-fa18a33175de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:37.935593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176aa27b-2881-4c6b-b17c-fa18a33175de None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:37.942191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaf21b5-92db-4c58-b45a-ee7bd931f4a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:37.942843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf21b5-92db-4c58-b45a-ee7bd931f4a9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:37.943132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf21b5-92db-4c58-b45a-ee7bd931f4a9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:37.943415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf21b5-92db-4c58-b45a-ee7bd931f4a9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:37.944347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-176aa27b-2881-4c6b-b17c-fa18a33175de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:37.953424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:37.954507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:37.955540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:37.955540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:37.955540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:37.955540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:37.955540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 322/646] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:37.955540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176aa27b-2881-4c6b-b17c-fa18a33175de None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:37.957108 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaf21b5-92db-4c58-b45a-ee7bd931f4a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-cdaf21b5-92db-4c58-b45a-ee7bd931f4a9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:37.962564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:37.963490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:37.964349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:37.968492 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 325/647] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:37.969360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaf21b5-92db-4c58-b45a-ee7bd931f4a9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.066688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176aa27b-2881-4c6b-b17c-fa18a33175de None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.075391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaf21b5-92db-4c58-b45a-ee7bd931f4a9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.100083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625378a0-26ae-4f16-9d9a-102880979a0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.100762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625378a0-26ae-4f16-9d9a-102880979a0a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.101055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625378a0-26ae-4f16-9d9a-102880979a0a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.101335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625378a0-26ae-4f16-9d9a-102880979a0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.108520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aecca77-035b-4bfd-9e7e-ab1d48e3471c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.109171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aecca77-035b-4bfd-9e7e-ab1d48e3471c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.109458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aecca77-035b-4bfd-9e7e-ab1d48e3471c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.109737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aecca77-035b-4bfd-9e7e-ab1d48e3471c None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.111032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aecca77-035b-4bfd-9e7e-ab1d48e3471c None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:38.111341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aecca77-035b-4bfd-9e7e-ab1d48e3471c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:38.112795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625378a0-26ae-4f16-9d9a-102880979a0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:38.112913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aecca77-035b-4bfd-9e7e-ab1d48e3471c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:38.114229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625378a0-26ae-4f16-9d9a-102880979a0a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:38.119210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:38.125676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 326/648] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:38.127295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aecca77-035b-4bfd-9e7e-ab1d48e3471c None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-625378a0-26ae-4f16-9d9a-102880979a0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:38.129726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:38.130745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:38.131639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:38.131639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:38.131639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:38.131639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:38.132758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 323/649] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:38.133582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625378a0-26ae-4f16-9d9a-102880979a0a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.186624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:38.230336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aecca77-035b-4bfd-9e7e-ab1d48e3471c None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.240143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625378a0-26ae-4f16-9d9a-102880979a0a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.263791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ccbddd-9195-4a8a-acb2-7aeff2d0a5b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.264403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ccbddd-9195-4a8a-acb2-7aeff2d0a5b1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.264715 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ccbddd-9195-4a8a-acb2-7aeff2d0a5b1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.264994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ccbddd-9195-4a8a-acb2-7aeff2d0a5b1 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.265682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ccbddd-9195-4a8a-acb2-7aeff2d0a5b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:38.265984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ccbddd-9195-4a8a-acb2-7aeff2d0a5b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:38.267373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ccbddd-9195-4a8a-acb2-7aeff2d0a5b1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:38.273058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f072330-8946-4ebd-b450-100012002651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.273674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f072330-8946-4ebd-b450-100012002651 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.273965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f072330-8946-4ebd-b450-100012002651 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.274239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f072330-8946-4ebd-b450-100012002651 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.274954 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f072330-8946-4ebd-b450-100012002651 None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:38.275263 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f072330-8946-4ebd-b450-100012002651 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:38.276625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f072330-8946-4ebd-b450-100012002651 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:38.278908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 327/650] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:38.280108 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84886ff-3593-405b-8a33-b0b2095cdfaf None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.280404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84886ff-3593-405b-8a33-b0b2095cdfaf None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.280718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84886ff-3593-405b-8a33-b0b2095cdfaf None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.284669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 328/651] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:38 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:38.284669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84886ff-3593-405b-8a33-b0b2095cdfaf None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.289946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 324/652] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:38.289946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f072330-8946-4ebd-b450-100012002651 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.342578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:38.385981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84886ff-3593-405b-8a33-b0b2095cdfaf None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.387764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f072330-8946-4ebd-b450-100012002651 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.400503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:38.420746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a5dbda-0807-4c56-815e-666f32a927ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.421393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a5dbda-0807-4c56-815e-666f32a927ca None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.421636 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14d1679-a1c7-45de-a05c-4a068fc83572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.422089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a5dbda-0807-4c56-815e-666f32a927ca None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.422089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a5dbda-0807-4c56-815e-666f32a927ca None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.422276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14d1679-a1c7-45de-a05c-4a068fc83572 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.422520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14d1679-a1c7-45de-a05c-4a068fc83572 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.423568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14d1679-a1c7-45de-a05c-4a068fc83572 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.424261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14d1679-a1c7-45de-a05c-4a068fc83572 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:38.424580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14d1679-a1c7-45de-a05c-4a068fc83572 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:38.425946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14d1679-a1c7-45de-a05c-4a068fc83572 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:38.429503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a5dbda-0807-4c56-815e-666f32a927ca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:38.430938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a5dbda-0807-4c56-815e-666f32a927ca None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:38.437343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 325/653] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:38.438132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14d1679-a1c7-45de-a05c-4a068fc83572 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-a3a5dbda-0807-4c56-815e-666f32a927ca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:38.442221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:38.443171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:38.444086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:38.444086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:38.444086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:38.444086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:38.444086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:38.444086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:38.444086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:38.444086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:38.444086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 329/654] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:38.444708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a9c65c-d4a8-427d-a0d6-f47674176f03 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.444791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a9c65c-d4a8-427d-a0d6-f47674176f03 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.445266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a9c65c-d4a8-427d-a0d6-f47674176f03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.462650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-22a9c65c-d4a8-427d-a0d6-f47674176f03 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:38.463841 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-22a9c65c-d4a8-427d-a0d6-f47674176f03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:38.528247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 330/655] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:38.529179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a9c65c-d4a8-427d-a0d6-f47674176f03 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.541941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14d1679-a1c7-45de-a05c-4a068fc83572 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.577303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accca03f-4963-4db9-a932-7fae1065457e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.577942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accca03f-4963-4db9-a932-7fae1065457e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.578231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accca03f-4963-4db9-a932-7fae1065457e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.578513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accca03f-4963-4db9-a932-7fae1065457e None admin] PATH_INFO: `/v3/users/glance` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.586552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accca03f-4963-4db9-a932-7fae1065457e None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:38.587998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-accca03f-4963-4db9-a932-7fae1065457e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:38.591760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-accca03f-4963-4db9-a932-7fae1065457e None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:38.600980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:38.601989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:38.602922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 326/656] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/users/glance => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:38.603520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accca03f-4963-4db9-a932-7fae1065457e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.634441 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:38.646907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a9c65c-d4a8-427d-a0d6-f47674176f03 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.683399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58961f5d-24df-4046-a58c-073a5b0c59cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.684047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58961f5d-24df-4046-a58c-073a5b0c59cb None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.684335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58961f5d-24df-4046-a58c-073a5b0c59cb None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.684642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58961f5d-24df-4046-a58c-073a5b0c59cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.695515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58961f5d-24df-4046-a58c-073a5b0c59cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:38.697076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58961f5d-24df-4046-a58c-073a5b0c59cb None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:38.707878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accca03f-4963-4db9-a932-7fae1065457e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-58961f5d-24df-4046-a58c-073a5b0c59cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:38.711755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:38.712635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:38.713540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:38.713540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:38.713540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:38.713540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:38.713540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 331/657] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:38.713810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58961f5d-24df-4046-a58c-073a5b0c59cb None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.744367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eabae15-3bfc-4668-93fc-b346acf00373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.745041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eabae15-3bfc-4668-93fc-b346acf00373 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.745326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eabae15-3bfc-4668-93fc-b346acf00373 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.745605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eabae15-3bfc-4668-93fc-b346acf00373 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.753024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eabae15-3bfc-4668-93fc-b346acf00373 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:38.754498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eabae15-3bfc-4668-93fc-b346acf00373 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-6eabae15-3bfc-4668-93fc-b346acf00373 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:38.765774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:38.766720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:38.767586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:38.767586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:38.767586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:38.767586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:38.767586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:38.767586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:38.767586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:38.767586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:38.767586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 327/658] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:38.768062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eabae15-3bfc-4668-93fc-b346acf00373 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.811755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:38.815549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58961f5d-24df-4046-a58c-073a5b0c59cb None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.848708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4cfe68-6501-4796-b7e2-06abd4e1b126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.849376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4cfe68-6501-4796-b7e2-06abd4e1b126 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.849664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4cfe68-6501-4796-b7e2-06abd4e1b126 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.849939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4cfe68-6501-4796-b7e2-06abd4e1b126 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.850693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4cfe68-6501-4796-b7e2-06abd4e1b126 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:38.851053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4cfe68-6501-4796-b7e2-06abd4e1b126 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:38.852433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4cfe68-6501-4796-b7e2-06abd4e1b126 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:38.859980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:38.864916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 332/659] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:38.865710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4cfe68-6501-4796-b7e2-06abd4e1b126 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.875019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eabae15-3bfc-4668-93fc-b346acf00373 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:38.908419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef38ce7-0daa-4c87-8b8f-6e5f30dfc7b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:38.909087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef38ce7-0daa-4c87-8b8f-6e5f30dfc7b1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:38.909379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef38ce7-0daa-4c87-8b8f-6e5f30dfc7b1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:38.909658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef38ce7-0daa-4c87-8b8f-6e5f30dfc7b1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:38.919764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fef38ce7-0daa-4c87-8b8f-6e5f30dfc7b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-fef38ce7-0daa-4c87-8b8f-6e5f30dfc7b1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:38.926165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:38.927088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:38.927903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 328/660] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:38.928639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef38ce7-0daa-4c87-8b8f-6e5f30dfc7b1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:38.968129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4cfe68-6501-4796-b7e2-06abd4e1b126 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.001859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6aeef2-c760-4a64-9c25-6d6d04fbdc06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.002477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6aeef2-c760-4a64-9c25-6d6d04fbdc06 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.002770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6aeef2-c760-4a64-9c25-6d6d04fbdc06 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.003055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6aeef2-c760-4a64-9c25-6d6d04fbdc06 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.003842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6aeef2-c760-4a64-9c25-6d6d04fbdc06 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:39.004155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6aeef2-c760-4a64-9c25-6d6d04fbdc06 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.005615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6aeef2-c760-4a64-9c25-6d6d04fbdc06 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:39.022091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 333/661] 10.222.0.40 () {56 vars in 1137 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:39.022883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6aeef2-c760-4a64-9c25-6d6d04fbdc06 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:39.027368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef38ce7-0daa-4c87-8b8f-6e5f30dfc7b1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.043510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:39.062269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b363f379-1ba7-4dc8-9025-d37d4978107d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.062891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b363f379-1ba7-4dc8-9025-d37d4978107d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.063177 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b363f379-1ba7-4dc8-9025-d37d4978107d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.063603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b363f379-1ba7-4dc8-9025-d37d4978107d None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.064300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b363f379-1ba7-4dc8-9025-d37d4978107d None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:39.064626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b363f379-1ba7-4dc8-9025-d37d4978107d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.066001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b363f379-1ba7-4dc8-9025-d37d4978107d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:39.077379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 329/662] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:39.078162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b363f379-1ba7-4dc8-9025-d37d4978107d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:39.091896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:39.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6aeef2-c760-4a64-9c25-6d6d04fbdc06 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.159989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21825c09-2615-44f9-9046-280baab22c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.160629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21825c09-2615-44f9-9046-280baab22c0b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.160911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21825c09-2615-44f9-9046-280baab22c0b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.161184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21825c09-2615-44f9-9046-280baab22c0b None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.161934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21825c09-2615-44f9-9046-280baab22c0b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:39.162241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21825c09-2615-44f9-9046-280baab22c0b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.163675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21825c09-2615-44f9-9046-280baab22c0b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:39.176047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 334/663] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:39.176884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21825c09-2615-44f9-9046-280baab22c0b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:39.183749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b363f379-1ba7-4dc8-9025-d37d4978107d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.220790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b699fd-1740-44bf-8509-722d95740933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.221724 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b699fd-1740-44bf-8509-722d95740933 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.222168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b699fd-1740-44bf-8509-722d95740933 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.222450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b699fd-1740-44bf-8509-722d95740933 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.223249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b699fd-1740-44bf-8509-722d95740933 None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=de49172e2a054096b319a7e4349ecaea) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:39.223569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b699fd-1740-44bf-8509-722d95740933 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.225025 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83b699fd-1740-44bf-8509-722d95740933 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:39.230813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 330/664] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:39 2022] GET /identity/v3/role_assignments?user.id=de49172e2a054096b319a7e4349ecaea&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:39.231594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b699fd-1740-44bf-8509-722d95740933 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:39.259276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:39.283947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21825c09-2615-44f9-9046-280baab22c0b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.322155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b893abb5-c6fc-4d02-8aa6-39fe4bdf1c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.322803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b893abb5-c6fc-4d02-8aa6-39fe4bdf1c9e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.323092 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b893abb5-c6fc-4d02-8aa6-39fe4bdf1c9e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.323381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b893abb5-c6fc-4d02-8aa6-39fe4bdf1c9e None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.324117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b893abb5-c6fc-4d02-8aa6-39fe4bdf1c9e None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:39.324948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b893abb5-c6fc-4d02-8aa6-39fe4bdf1c9e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.325871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b893abb5-c6fc-4d02-8aa6-39fe4bdf1c9e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:39.335779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 335/665] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:39.336549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b893abb5-c6fc-4d02-8aa6-39fe4bdf1c9e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:39.342324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:39.371872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b699fd-1740-44bf-8509-722d95740933 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.412072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b043bc-c97d-4aa4-b95b-44ffe596a8aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.413315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b043bc-c97d-4aa4-b95b-44ffe596a8aa None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.413607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b043bc-c97d-4aa4-b95b-44ffe596a8aa None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.414671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b043bc-c97d-4aa4-b95b-44ffe596a8aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.451641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b893abb5-c6fc-4d02-8aa6-39fe4bdf1c9e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b043bc-c97d-4aa4-b95b-44ffe596a8aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.489844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b043bc-c97d-4aa4-b95b-44ffe596a8aa None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:39.490753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4485b738-f592-4ea0-8354-79bec58f3615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.491410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4485b738-f592-4ea0-8354-79bec58f3615 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.491869 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4485b738-f592-4ea0-8354-79bec58f3615 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.492160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4485b738-f592-4ea0-8354-79bec58f3615 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.504029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4485b738-f592-4ea0-8354-79bec58f3615 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.506359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4485b738-f592-4ea0-8354-79bec58f3615 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:39.510084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-4485b738-f592-4ea0-8354-79bec58f3615 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:39.524371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:39.525389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 336/666] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:39 2022] GET /identity/v3/users/neutron => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:39.531436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4485b738-f592-4ea0-8354-79bec58f3615 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:39.570200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:39.594990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 331/667] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:39.595981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b043bc-c97d-4aa4-b95b-44ffe596a8aa None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:39.648008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4485b738-f592-4ea0-8354-79bec58f3615 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.688156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178ed3dc-3bb0-4380-86df-638659e8c105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.689899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178ed3dc-3bb0-4380-86df-638659e8c105 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.690192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178ed3dc-3bb0-4380-86df-638659e8c105 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.690472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178ed3dc-3bb0-4380-86df-638659e8c105 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.691271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178ed3dc-3bb0-4380-86df-638659e8c105 None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=58c33a72a871432b88da4f0bf1bc514c) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:39.691586 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178ed3dc-3bb0-4380-86df-638659e8c105 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.693144 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178ed3dc-3bb0-4380-86df-638659e8c105 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:39.699416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 337/668] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:39 2022] GET /identity/v3/role_assignments?user.id=58c33a72a871432b88da4f0bf1bc514c&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:39.705635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178ed3dc-3bb0-4380-86df-638659e8c105 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:39.709825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b043bc-c97d-4aa4-b95b-44ffe596a8aa None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.749434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660f5b08-3dd8-4eec-940c-525e84ff73f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.750193 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660f5b08-3dd8-4eec-940c-525e84ff73f9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.750554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660f5b08-3dd8-4eec-940c-525e84ff73f9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.751433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660f5b08-3dd8-4eec-940c-525e84ff73f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.764367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660f5b08-3dd8-4eec-940c-525e84ff73f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.766057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660f5b08-3dd8-4eec-940c-525e84ff73f9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-660f5b08-3dd8-4eec-940c-525e84ff73f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:39.784312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:39.785560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:39.786678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:39.786678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:39.786678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:39.786678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:39.792264 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 332/669] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:39.793276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:39.793403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660f5b08-3dd8-4eec-940c-525e84ff73f9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:39.835993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:39.844445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178ed3dc-3bb0-4380-86df-638659e8c105 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.893681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fac80d0-264d-437d-9056-8b6b05cca54f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.894910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fac80d0-264d-437d-9056-8b6b05cca54f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.895201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fac80d0-264d-437d-9056-8b6b05cca54f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.895486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fac80d0-264d-437d-9056-8b6b05cca54f None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.896678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fac80d0-264d-437d-9056-8b6b05cca54f None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:39.897008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fac80d0-264d-437d-9056-8b6b05cca54f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.899079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fac80d0-264d-437d-9056-8b6b05cca54f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:39.922502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 338/670] 10.222.0.40 () {56 vars in 1139 bytes} [Fri Jul 29 04:34:39 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:39.923794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fac80d0-264d-437d-9056-8b6b05cca54f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:39.926272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660f5b08-3dd8-4eec-940c-525e84ff73f9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:39.968144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c321ee5f-e242-4bf9-be8a-278c68e744ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:39.968757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c321ee5f-e242-4bf9-be8a-278c68e744ff None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:39.969046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c321ee5f-e242-4bf9-be8a-278c68e744ff None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:39.969325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c321ee5f-e242-4bf9-be8a-278c68e744ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:39.981792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c321ee5f-e242-4bf9-be8a-278c68e744ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:39.983312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c321ee5f-e242-4bf9-be8a-278c68e744ff None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-c321ee5f-e242-4bf9-be8a-278c68e744ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:40.002138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:40.003241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:40.004537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:40.004537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:40.004537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:40.004537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:40.004537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 333/671] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:40.004537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c321ee5f-e242-4bf9-be8a-278c68e744ff None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:40.068847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fac80d0-264d-437d-9056-8b6b05cca54f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:40.096153 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:40.103298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:40.117750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c321ee5f-e242-4bf9-be8a-278c68e744ff None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:40.121986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396228e5-c5f3-40c6-84c9-45640f4ded2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:40.122619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396228e5-c5f3-40c6-84c9-45640f4ded2a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:40.122908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396228e5-c5f3-40c6-84c9-45640f4ded2a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:40.123191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396228e5-c5f3-40c6-84c9-45640f4ded2a None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:40.123899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396228e5-c5f3-40c6-84c9-45640f4ded2a None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:40.124209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396228e5-c5f3-40c6-84c9-45640f4ded2a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:40.125645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396228e5-c5f3-40c6-84c9-45640f4ded2a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:40.138780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 339/672] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:40.139624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396228e5-c5f3-40c6-84c9-45640f4ded2a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:40.162537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fee64ed-1d98-4129-bcff-b230775cdbee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:40.163734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fee64ed-1d98-4129-bcff-b230775cdbee None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:40.164215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fee64ed-1d98-4129-bcff-b230775cdbee None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:40.164961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fee64ed-1d98-4129-bcff-b230775cdbee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:40.254417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396228e5-c5f3-40c6-84c9-45640f4ded2a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:40.264562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fee64ed-1d98-4129-bcff-b230775cdbee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:40.267523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fee64ed-1d98-4129-bcff-b230775cdbee None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:40.291078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3cbdde-4e0f-4b76-bb85-84a5aa0dfca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:40.291823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3cbdde-4e0f-4b76-bb85-84a5aa0dfca7 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:40.292252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3cbdde-4e0f-4b76-bb85-84a5aa0dfca7 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:40.292563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3cbdde-4e0f-4b76-bb85-84a5aa0dfca7 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:40.293382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3cbdde-4e0f-4b76-bb85-84a5aa0dfca7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:40.293879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3cbdde-4e0f-4b76-bb85-84a5aa0dfca7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:40.295345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3cbdde-4e0f-4b76-bb85-84a5aa0dfca7 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:40.315331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 340/673] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:40.316340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3cbdde-4e0f-4b76-bb85-84a5aa0dfca7 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:40.381668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:40.383527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:40.414132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 334/674] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:40.414132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fee64ed-1d98-4129-bcff-b230775cdbee None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:40.429617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3cbdde-4e0f-4b76-bb85-84a5aa0dfca7 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:40.471022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b8b53e-8acb-4807-abfe-46e9ceb6ebd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:40.471720 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b8b53e-8acb-4807-abfe-46e9ceb6ebd2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:40.472006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b8b53e-8acb-4807-abfe-46e9ceb6ebd2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:40.472339 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b8b53e-8acb-4807-abfe-46e9ceb6ebd2 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:40.480331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b8b53e-8acb-4807-abfe-46e9ceb6ebd2 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:40.481823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b8b53e-8acb-4807-abfe-46e9ceb6ebd2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-91b8b53e-8acb-4807-abfe-46e9ceb6ebd2 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:40.494425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:40.495418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:40.496347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:40.496347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:40.496347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:40.496347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:40.496347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:40.496347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:40.496347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:40.496347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:40.496347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:40.496347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:40.500244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 341/675] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:40 2022] GET /identity/v3/users/swift => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:40.501629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b8b53e-8acb-4807-abfe-46e9ceb6ebd2 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:40.546228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fee64ed-1d98-4129-bcff-b230775cdbee None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:40.588115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa0648a-296c-4b47-916f-7efe95a7a899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:40.589150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa0648a-296c-4b47-916f-7efe95a7a899 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:40.589907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa0648a-296c-4b47-916f-7efe95a7a899 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:40.591274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa0648a-296c-4b47-916f-7efe95a7a899 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:40.600727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa0648a-296c-4b47-916f-7efe95a7a899 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:40.602925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa0648a-296c-4b47-916f-7efe95a7a899 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-cfa0648a-296c-4b47-916f-7efe95a7a899 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:40.617046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:40.617945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:40.618834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:40.618834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:40.618834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:40.618834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:40.618834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:40.618834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:40.618834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:40.618834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:40.623165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 335/676] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:40.623998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa0648a-296c-4b47-916f-7efe95a7a899 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:40.631937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b8b53e-8acb-4807-abfe-46e9ceb6ebd2 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:40.633566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:40.684222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:40.692930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8e02c1-d06c-4c31-aaa4-fc5bd6225d35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:40.694768 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8e02c1-d06c-4c31-aaa4-fc5bd6225d35 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:40.695170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8e02c1-d06c-4c31-aaa4-fc5bd6225d35 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:40.695547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8e02c1-d06c-4c31-aaa4-fc5bd6225d35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:40.709367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8e02c1-d06c-4c31-aaa4-fc5bd6225d35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:40.711012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8e02c1-d06c-4c31-aaa4-fc5bd6225d35 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-8f8e02c1-d06c-4c31-aaa4-fc5bd6225d35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:40.727495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:40.728624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:40.729598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:40.729598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:40.729598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:40.729598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:40.731253 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 342/677] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:40.732229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8e02c1-d06c-4c31-aaa4-fc5bd6225d35 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:40.759035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa0648a-296c-4b47-916f-7efe95a7a899 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:40.797020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4770076d-783e-4b4a-9481-b694f744ce58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:40.798831 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4770076d-783e-4b4a-9481-b694f744ce58 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:40.799330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4770076d-783e-4b4a-9481-b694f744ce58 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:40.799820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4770076d-783e-4b4a-9481-b694f744ce58 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:40.800852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4770076d-783e-4b4a-9481-b694f744ce58 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:40.801371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4770076d-783e-4b4a-9481-b694f744ce58 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:40.803077 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4770076d-783e-4b4a-9481-b694f744ce58 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:40.826408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 336/678] 10.222.0.40 () {56 vars in 1135 bytes} [Fri Jul 29 04:34:40 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:40.827601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4770076d-783e-4b4a-9481-b694f744ce58 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:40.863165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8e02c1-d06c-4c31-aaa4-fc5bd6225d35 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:40.901577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0dbc3-6acb-4718-8d80-665e3e160219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:40.902329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0dbc3-6acb-4718-8d80-665e3e160219 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:40.902616 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0dbc3-6acb-4718-8d80-665e3e160219 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:40.902900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c0dbc3-6acb-4718-8d80-665e3e160219 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:40.904367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c0dbc3-6acb-4718-8d80-665e3e160219 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:40.904367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c0dbc3-6acb-4718-8d80-665e3e160219 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:40.905525 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c0dbc3-6acb-4718-8d80-665e3e160219 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:40.925433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 343/679] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:40.925433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0dbc3-6acb-4718-8d80-665e3e160219 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:40.940127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:40.941720 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:40.944958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4770076d-783e-4b4a-9481-b694f744ce58 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:40.983914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5350be94-fef6-431d-aac5-d434032cfc56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:40.984681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5350be94-fef6-431d-aac5-d434032cfc56 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:40.985601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5350be94-fef6-431d-aac5-d434032cfc56 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:40.986296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5350be94-fef6-431d-aac5-d434032cfc56 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:40.986972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5350be94-fef6-431d-aac5-d434032cfc56 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:40.987359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5350be94-fef6-431d-aac5-d434032cfc56 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:40.989120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5350be94-fef6-431d-aac5-d434032cfc56 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:41.006406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 337/680] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:41.006406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5350be94-fef6-431d-aac5-d434032cfc56 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:41.046395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c0dbc3-6acb-4718-8d80-665e3e160219 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:41.086991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66226ff-31e6-4638-a692-c6f42bc4bee2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:41.087940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66226ff-31e6-4638-a692-c6f42bc4bee2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:41.088428 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66226ff-31e6-4638-a692-c6f42bc4bee2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:41.089028 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66226ff-31e6-4638-a692-c6f42bc4bee2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:41.113509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5350be94-fef6-431d-aac5-d434032cfc56 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:41.149136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a16c59-247b-457b-8e76-082d950aed54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:41.149818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a16c59-247b-457b-8e76-082d950aed54 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:41.150170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a16c59-247b-457b-8e76-082d950aed54 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:41.150456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a16c59-247b-457b-8e76-082d950aed54 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:41.151301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a16c59-247b-457b-8e76-082d950aed54 None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=3749dc021aac478f88072ced48f4cee4) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:41.151619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a16c59-247b-457b-8e76-082d950aed54 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:41.153167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a16c59-247b-457b-8e76-082d950aed54 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:41.164933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 338/681] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:41 2022] GET /identity/v3/role_assignments?user.id=3749dc021aac478f88072ced48f4cee4&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:41.165900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a16c59-247b-457b-8e76-082d950aed54 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:41.169589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66226ff-31e6-4638-a692-c6f42bc4bee2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:41.171455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66226ff-31e6-4638-a692-c6f42bc4bee2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:41.191840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:41.206400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:41.302672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 344/682] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:41.304436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66226ff-31e6-4638-a692-c6f42bc4bee2 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:41.337967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a16c59-247b-457b-8e76-082d950aed54 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:41.375348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b4bfaf-d1dc-4cc7-931c-9235ba2f0a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:41.376741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b4bfaf-d1dc-4cc7-931c-9235ba2f0a75 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:41.376921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b4bfaf-d1dc-4cc7-931c-9235ba2f0a75 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:41.377181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b4bfaf-d1dc-4cc7-931c-9235ba2f0a75 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:41.384866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b4bfaf-d1dc-4cc7-931c-9235ba2f0a75 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:41.386354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b4bfaf-d1dc-4cc7-931c-9235ba2f0a75 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-f6b4bfaf-d1dc-4cc7-931c-9235ba2f0a75 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:41.400370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:41.401357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:41.402229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:41.402229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:41.402229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:41.402229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:41.402229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:41.402229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:41.402229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:41.402229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:41.404691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 339/683] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:41.406358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f116cb8-3176-457f-8dca-85d19eda22a1 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:41.406784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f116cb8-3176-457f-8dca-85d19eda22a1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:41.407960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f116cb8-3176-457f-8dca-85d19eda22a1 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:41.409628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 340/684] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:41.410860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f116cb8-3176-457f-8dca-85d19eda22a1 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:41.443756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66226ff-31e6-4638-a692-c6f42bc4bee2 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:41.463761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:41.491483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905f72f2-8713-4c20-924d-e1e22c47720e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:41.492287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905f72f2-8713-4c20-924d-e1e22c47720e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:41.492576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905f72f2-8713-4c20-924d-e1e22c47720e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:41.492927 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905f72f2-8713-4c20-924d-e1e22c47720e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:41.505470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905f72f2-8713-4c20-924d-e1e22c47720e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:41.506171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905f72f2-8713-4c20-924d-e1e22c47720e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-905f72f2-8713-4c20-924d-e1e22c47720e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:41.524221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:41.525403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:41.526297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:41.526297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:41.526297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:41.526297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:41.526297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 345/685] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:41.526984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b893c-d2cf-4030-bd00-bf6a70d43070 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:41.527278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b893c-d2cf-4030-bd00-bf6a70d43070 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:41.527560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4b893c-d2cf-4030-bd00-bf6a70d43070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:41.529441 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:41.551963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f116cb8-3176-457f-8dca-85d19eda22a1 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:41.555648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6c4b893c-d2cf-4030-bd00-bf6a70d43070 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:41.556932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6c4b893c-d2cf-4030-bd00-bf6a70d43070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:41.602518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c127ba2-826b-4e90-8126-9cee3c4584af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:41.603070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c127ba2-826b-4e90-8126-9cee3c4584af None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:41.603348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c127ba2-826b-4e90-8126-9cee3c4584af None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:41.603629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c127ba2-826b-4e90-8126-9cee3c4584af None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:41.604393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c127ba2-826b-4e90-8126-9cee3c4584af None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:41.604734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c127ba2-826b-4e90-8126-9cee3c4584af None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:41.606180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c127ba2-826b-4e90-8126-9cee3c4584af None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:41.619819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 341/686] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:41.625546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c127ba2-826b-4e90-8126-9cee3c4584af None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:41.651963 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 346/687] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:41.653172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4b893c-d2cf-4030-bd00-bf6a70d43070 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:41.772284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:41.772734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c127ba2-826b-4e90-8126-9cee3c4584af None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:41.778974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4b893c-d2cf-4030-bd00-bf6a70d43070 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:41.817622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a74098-c77a-4111-9299-b9b0ad451e98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:41.818409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a74098-c77a-4111-9299-b9b0ad451e98 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:41.818790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a74098-c77a-4111-9299-b9b0ad451e98 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:41.819159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a74098-c77a-4111-9299-b9b0ad451e98 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:41.820064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a74098-c77a-4111-9299-b9b0ad451e98 None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=f6726ef0c4184040a18a76898bb738d1) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:41.820482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a74098-c77a-4111-9299-b9b0ad451e98 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:41.822183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a74098-c77a-4111-9299-b9b0ad451e98 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:41.823486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459be505-3c66-4852-ad94-e54bb61fb867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:41.824551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459be505-3c66-4852-ad94-e54bb61fb867 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:41.825014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459be505-3c66-4852-ad94-e54bb61fb867 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:41.825993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459be505-3c66-4852-ad94-e54bb61fb867 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:41.826854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459be505-3c66-4852-ad94-e54bb61fb867 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:41.827594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459be505-3c66-4852-ad94-e54bb61fb867 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:41.829176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459be505-3c66-4852-ad94-e54bb61fb867 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:41.833034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 347/688] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:41 2022] GET /identity/v3/role_assignments?user.id=f6726ef0c4184040a18a76898bb738d1&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:41.834039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a74098-c77a-4111-9299-b9b0ad451e98 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:41.849088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 342/689] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:41.850975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459be505-3c66-4852-ad94-e54bb61fb867 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:41.854127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:41.970237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a74098-c77a-4111-9299-b9b0ad451e98 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:42.010023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb1531a-0fe9-484e-8a7f-6f040a012f7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:42.010668 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb1531a-0fe9-484e-8a7f-6f040a012f7e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.010958 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb1531a-0fe9-484e-8a7f-6f040a012f7e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.011241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb1531a-0fe9-484e-8a7f-6f040a012f7e None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.011881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb1531a-0fe9-484e-8a7f-6f040a012f7e None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:42.012871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459be505-3c66-4852-ad94-e54bb61fb867 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:42.013307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb1531a-0fe9-484e-8a7f-6f040a012f7e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-abb1531a-0fe9-484e-8a7f-6f040a012f7e None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:42.025626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:42.026846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:42.027871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 348/690] 10.222.0.40 () {56 vars in 1129 bytes} [Fri Jul 29 04:34:41 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:42.029027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee6f57-1444-4c2c-9caf-1a9df58ec13f None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.029027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee6f57-1444-4c2c-9caf-1a9df58ec13f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.029027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee6f57-1444-4c2c-9caf-1a9df58ec13f None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.030363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 349/691] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:42.036152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2db989-e45f-4c5e-97fa-1c8700693e46 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.036665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2db989-e45f-4c5e-97fa-1c8700693e46 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.037096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2db989-e45f-4c5e-97fa-1c8700693e46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.056307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d52ad3-b678-41a4-9d7f-fa87c3e2a0af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:42.057760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d52ad3-b678-41a4-9d7f-fa87c3e2a0af None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.057873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4a2db989-e45f-4c5e-97fa-1c8700693e46 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:42.058044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d52ad3-b678-41a4-9d7f-fa87c3e2a0af None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.058368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d52ad3-b678-41a4-9d7f-fa87c3e2a0af None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.059175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4a2db989-e45f-4c5e-97fa-1c8700693e46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:42.065836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d52ad3-b678-41a4-9d7f-fa87c3e2a0af None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:42.067308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d52ad3-b678-41a4-9d7f-fa87c3e2a0af None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:42.080244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-46d52ad3-b678-41a4-9d7f-fa87c3e2a0af None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:42.081924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:42.082878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:42.083777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:42.083777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:42.083777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:42.083777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:42.083777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:42.083777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:42.083777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:42.083777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:42.083777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 343/692] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:42.084276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d52ad3-b678-41a4-9d7f-fa87c3e2a0af None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:42.145539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 350/693] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:34:42.146782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2db989-e45f-4c5e-97fa-1c8700693e46 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:42.177446 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:42.230553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d52ad3-b678-41a4-9d7f-fa87c3e2a0af None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:42.273308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbb08a1-2ca6-40d0-bcb2-e889b9076c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:42.274431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbb08a1-2ca6-40d0-bcb2-e889b9076c44 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.275287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbb08a1-2ca6-40d0-bcb2-e889b9076c44 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.275998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbb08a1-2ca6-40d0-bcb2-e889b9076c44 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.276854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbb08a1-2ca6-40d0-bcb2-e889b9076c44 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:42.277309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbb08a1-2ca6-40d0-bcb2-e889b9076c44 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:42.277845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2db989-e45f-4c5e-97fa-1c8700693e46 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:42.279038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbb08a1-2ca6-40d0-bcb2-e889b9076c44 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:42.294305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 344/694] 10.222.0.40 () {56 vars in 1124 bytes} [Fri Jul 29 04:34:42 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:42.294305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbb08a1-2ca6-40d0-bcb2-e889b9076c44 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:42.330315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00451242-682e-4969-8a2d-d6a321f2bd0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:42.331561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00451242-682e-4969-8a2d-d6a321f2bd0d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.332114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00451242-682e-4969-8a2d-d6a321f2bd0d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.333078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00451242-682e-4969-8a2d-d6a321f2bd0d None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.333757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00451242-682e-4969-8a2d-d6a321f2bd0d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:42.334074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00451242-682e-4969-8a2d-d6a321f2bd0d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:42.335551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00451242-682e-4969-8a2d-d6a321f2bd0d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:42.355341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 351/695] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:42.356463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00451242-682e-4969-8a2d-d6a321f2bd0d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:42.378732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:42.420294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbb08a1-2ca6-40d0-bcb2-e889b9076c44 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:42.447338 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:42.463262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd105e-cb23-4cbc-8f94-1ca73581a943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:42.463907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd105e-cb23-4cbc-8f94-1ca73581a943 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.464189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd105e-cb23-4cbc-8f94-1ca73581a943 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.464461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd105e-cb23-4cbc-8f94-1ca73581a943 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.477164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bd105e-cb23-4cbc-8f94-1ca73581a943 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-b1bd105e-cb23-4cbc-8f94-1ca73581a943 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:42.482190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:42.483112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:42.483988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:42.488894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 345/696] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:42.489772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd105e-cb23-4cbc-8f94-1ca73581a943 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:42.490873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00451242-682e-4969-8a2d-d6a321f2bd0d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:42.537222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b446ec-8a7f-4c8d-b6ad-dfb5ed3f3485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:42.538449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b446ec-8a7f-4c8d-b6ad-dfb5ed3f3485 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.539249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b446ec-8a7f-4c8d-b6ad-dfb5ed3f3485 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.539686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b446ec-8a7f-4c8d-b6ad-dfb5ed3f3485 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.540810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b446ec-8a7f-4c8d-b6ad-dfb5ed3f3485 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:42.541277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b446ec-8a7f-4c8d-b6ad-dfb5ed3f3485 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:42.543251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b446ec-8a7f-4c8d-b6ad-dfb5ed3f3485 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:42.552728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 352/697] 10.222.0.40 () {56 vars in 1124 bytes} [Fri Jul 29 04:34:42 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:42.554420 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b446ec-8a7f-4c8d-b6ad-dfb5ed3f3485 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:42.647949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd105e-cb23-4cbc-8f94-1ca73581a943 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:42.660933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:42.692280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b446ec-8a7f-4c8d-b6ad-dfb5ed3f3485 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:42.733273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84166f76-5735-4ef4-9b7e-432b36c0b146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:42.734611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84166f76-5735-4ef4-9b7e-432b36c0b146 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.735237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84166f76-5735-4ef4-9b7e-432b36c0b146 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.735869 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84166f76-5735-4ef4-9b7e-432b36c0b146 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.737017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84166f76-5735-4ef4-9b7e-432b36c0b146 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:42.737484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84166f76-5735-4ef4-9b7e-432b36c0b146 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:42.739159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84166f76-5735-4ef4-9b7e-432b36c0b146 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:42.747086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa10826-5ad4-47fb-81c4-19d44d6ee83b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:42.747845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa10826-5ad4-47fb-81c4-19d44d6ee83b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.748229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa10826-5ad4-47fb-81c4-19d44d6ee83b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.748646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa10826-5ad4-47fb-81c4-19d44d6ee83b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.749610 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa10826-5ad4-47fb-81c4-19d44d6ee83b None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=bbfe92babca048d882cb59cb29b3ed97) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:42.749994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa10826-5ad4-47fb-81c4-19d44d6ee83b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:42.751551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 353/698] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:42.757141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84166f76-5735-4ef4-9b7e-432b36c0b146 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:42.757550 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa10826-5ad4-47fb-81c4-19d44d6ee83b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:42.771964 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 346/699] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:42 2022] GET /identity/v3/role_assignments?user.id=bbfe92babca048d882cb59cb29b3ed97&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 608 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:42.878379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84166f76-5735-4ef4-9b7e-432b36c0b146 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:42.918526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17b04d3-5713-4e62-bcf2-1155bac11c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:42.919571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17b04d3-5713-4e62-bcf2-1155bac11c1a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:42.920085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17b04d3-5713-4e62-bcf2-1155bac11c1a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:42.920573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17b04d3-5713-4e62-bcf2-1155bac11c1a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:42.932719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:42.936319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17b04d3-5713-4e62-bcf2-1155bac11c1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:42.938729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17b04d3-5713-4e62-bcf2-1155bac11c1a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-f17b04d3-5713-4e62-bcf2-1155bac11c1a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:42.956926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:42.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:42.958710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:42.958710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:42.958710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:42.958710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:42.960911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 354/700] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:42.966331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa10826-5ad4-47fb-81c4-19d44d6ee83b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:43.021724 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:43.077056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa10826-5ad4-47fb-81c4-19d44d6ee83b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:43.115928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496c275-0f2e-4aee-bfda-1e6bfcc9c009 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:43.116965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496c275-0f2e-4aee-bfda-1e6bfcc9c009 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:43.117487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496c275-0f2e-4aee-bfda-1e6bfcc9c009 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:43.117970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2496c275-0f2e-4aee-bfda-1e6bfcc9c009 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:43.118883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496c275-0f2e-4aee-bfda-1e6bfcc9c009 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:43.119397 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496c275-0f2e-4aee-bfda-1e6bfcc9c009 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:43.121001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2496c275-0f2e-4aee-bfda-1e6bfcc9c009 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:43.135674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 347/701] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:43.140725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17b04d3-5713-4e62-bcf2-1155bac11c1a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:43.266986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17b04d3-5713-4e62-bcf2-1155bac11c1a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:43.304555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4a14c2-b515-4553-9216-8ebe0e9eb1c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:43.305587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4a14c2-b515-4553-9216-8ebe0e9eb1c6 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:43.306104 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4a14c2-b515-4553-9216-8ebe0e9eb1c6 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:43.306591 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4a14c2-b515-4553-9216-8ebe0e9eb1c6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:43.432281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4a14c2-b515-4553-9216-8ebe0e9eb1c6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:43.434131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4a14c2-b515-4553-9216-8ebe0e9eb1c6 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-aa4a14c2-b515-4553-9216-8ebe0e9eb1c6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:43.447329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:43.448307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:43.449414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:43.449414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:43.449414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:43.449414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:43.449414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:43.449414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:43.449414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:43.449414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:43.449414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:43.449414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:43.452297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 355/702] 10.222.0.40 () {56 vars in 1093 bytes} [Fri Jul 29 04:34:43 2022] GET /identity/v3/users/nova => generated 82 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:43.457653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496c275-0f2e-4aee-bfda-1e6bfcc9c009 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:43.577737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2496c275-0f2e-4aee-bfda-1e6bfcc9c009 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:43.591081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:43.616410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bc413b-9fe1-48d0-bd06-bc76e4b29178 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:43.617068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bc413b-9fe1-48d0-bd06-bc76e4b29178 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:43.617354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bc413b-9fe1-48d0-bd06-bc76e4b29178 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:43.617632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bc413b-9fe1-48d0-bd06-bc76e4b29178 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:43.618440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bc413b-9fe1-48d0-bd06-bc76e4b29178 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:43.618752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bc413b-9fe1-48d0-bd06-bc76e4b29178 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:43.620262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bc413b-9fe1-48d0-bd06-bc76e4b29178 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:43.622208 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76494f31-fa95-4df6-a750-7f8019c80a8c None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:43.622493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76494f31-fa95-4df6-a750-7f8019c80a8c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:43.622771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76494f31-fa95-4df6-a750-7f8019c80a8c None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:43.624227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 356/703] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:43.629933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd82934-f254-4e35-bfc8-cbbf374976c9 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:43.630490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd82934-f254-4e35-bfc8-cbbf374976c9 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:43.630968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd82934-f254-4e35-bfc8-cbbf374976c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:43.643336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 348/704] 10.222.0.40 () {56 vars in 1133 bytes} [Fri Jul 29 04:34:43 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:43.651233 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bc413b-9fe1-48d0-bd06-bc76e4b29178 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:43.658477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0fd82934-f254-4e35-bfc8-cbbf374976c9 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:43.659931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0fd82934-f254-4e35-bfc8-cbbf374976c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:43.729828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:43.770626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 357/705] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:34:43.825718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bc413b-9fe1-48d0-bd06-bc76e4b29178 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:43.829473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd82934-f254-4e35-bfc8-cbbf374976c9 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:43.867019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbbd280-7c45-409a-8228-f7e130b3acaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:43.868515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbbd280-7c45-409a-8228-f7e130b3acaa None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:43.869035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbbd280-7c45-409a-8228-f7e130b3acaa None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:43.869540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbbd280-7c45-409a-8228-f7e130b3acaa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:43.928286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:43.970268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbbd280-7c45-409a-8228-f7e130b3acaa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:43.972075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbbd280-7c45-409a-8228-f7e130b3acaa None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:44.088460 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 349/706] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:44.090252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd82934-f254-4e35-bfc8-cbbf374976c9 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:44.097612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbbd280-7c45-409a-8228-f7e130b3acaa None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:44.142043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a71be8-81d1-4024-9d26-e8f24ed1b9b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:44.143526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a71be8-81d1-4024-9d26-e8f24ed1b9b7 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:44.144194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a71be8-81d1-4024-9d26-e8f24ed1b9b7 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:44.144647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a71be8-81d1-4024-9d26-e8f24ed1b9b7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:44.163855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a71be8-81d1-4024-9d26-e8f24ed1b9b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:44.170068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a71be8-81d1-4024-9d26-e8f24ed1b9b7 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-d3a71be8-81d1-4024-9d26-e8f24ed1b9b7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:44.191597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:44.192859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:44.193902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:44.193902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:44.193902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:44.193902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:44.193902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 358/707] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 365 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:44.194786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7c129b-4d9d-4060-b743-b6fb4dea687a None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:44.195227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7c129b-4d9d-4060-b743-b6fb4dea687a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:44.195338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7c129b-4d9d-4060-b743-b6fb4dea687a None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:44.201128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 359/708] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:44.202638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7c129b-4d9d-4060-b743-b6fb4dea687a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:44.211373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:44.253264 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:44.253670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbbd280-7c45-409a-8228-f7e130b3acaa None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:44.294959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5572169-f63f-4a01-96b3-8b1aaa3402bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:44.295901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5572169-f63f-4a01-96b3-8b1aaa3402bc None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:44.296416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5572169-f63f-4a01-96b3-8b1aaa3402bc None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:44.296877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5572169-f63f-4a01-96b3-8b1aaa3402bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:44.312185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5572169-f63f-4a01-96b3-8b1aaa3402bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:44.314450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5572169-f63f-4a01-96b3-8b1aaa3402bc None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-c5572169-f63f-4a01-96b3-8b1aaa3402bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:44.333813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:44.334758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:44.335634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:44.335634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:44.335634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:44.335634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:44.335634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 350/709] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:44.336716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baed209-a8cd-4836-9258-771be05fe945 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:44.337013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baed209-a8cd-4836-9258-771be05fe945 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:44.337128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7c129b-4d9d-4060-b743-b6fb4dea687a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:44.337289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baed209-a8cd-4836-9258-771be05fe945 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:44.357198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7baed209-a8cd-4836-9258-771be05fe945 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:44.358568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7baed209-a8cd-4836-9258-771be05fe945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:44.372807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1734f8ae-c066-410a-bd82-e27bd6881e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:44.373775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1734f8ae-c066-410a-bd82-e27bd6881e60 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:44.374290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1734f8ae-c066-410a-bd82-e27bd6881e60 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:44.374756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1734f8ae-c066-410a-bd82-e27bd6881e60 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:44.376225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1734f8ae-c066-410a-bd82-e27bd6881e60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:44.376806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1734f8ae-c066-410a-bd82-e27bd6881e60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:44.378388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1734f8ae-c066-410a-bd82-e27bd6881e60 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:44.394200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 360/710] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:44.395513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1734f8ae-c066-410a-bd82-e27bd6881e60 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:44.437333 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 351/711] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:44.439053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baed209-a8cd-4836-9258-771be05fe945 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:44.492269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:44.504918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1734f8ae-c066-410a-bd82-e27bd6881e60 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:44.520992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:44.541284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292717cf-0622-480d-a271-0d9f94eb5fd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:44.541944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292717cf-0622-480d-a271-0d9f94eb5fd2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:44.542304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292717cf-0622-480d-a271-0d9f94eb5fd2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:44.542658 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292717cf-0622-480d-a271-0d9f94eb5fd2 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:44.543438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292717cf-0622-480d-a271-0d9f94eb5fd2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:44.543876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292717cf-0622-480d-a271-0d9f94eb5fd2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:44.545323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292717cf-0622-480d-a271-0d9f94eb5fd2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:44.550135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baed209-a8cd-4836-9258-771be05fe945 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:44.558862 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 361/712] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:44.559912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292717cf-0622-480d-a271-0d9f94eb5fd2 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:44.591437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01e63bc-95ac-4422-95f7-f1f68e34381c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:44.592367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01e63bc-95ac-4422-95f7-f1f68e34381c None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:44.592933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01e63bc-95ac-4422-95f7-f1f68e34381c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:44.593419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01e63bc-95ac-4422-95f7-f1f68e34381c None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:44.594471 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01e63bc-95ac-4422-95f7-f1f68e34381c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:44.596115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01e63bc-95ac-4422-95f7-f1f68e34381c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:44.621026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 352/713] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:34:44 2022] POST /identity/v3/users => generated 254 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:34:44.622504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01e63bc-95ac-4422-95f7-f1f68e34381c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:44.700064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292717cf-0622-480d-a271-0d9f94eb5fd2 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:44.779016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5db4891-6bd4-4cdd-8e17-979a03bf2160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:44.780846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5db4891-6bd4-4cdd-8e17-979a03bf2160 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:44.781113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5db4891-6bd4-4cdd-8e17-979a03bf2160 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:44.781398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5db4891-6bd4-4cdd-8e17-979a03bf2160 None admin] PATH_INFO: `/v3/services/network` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:44.782089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5db4891-6bd4-4cdd-8e17-979a03bf2160 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:44.783496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5db4891-6bd4-4cdd-8e17-979a03bf2160 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c5db4891-6bd4-4cdd-8e17-979a03bf2160 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:44.802462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:44.803652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:44.804485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 362/714] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:44 2022] GET /identity/v3/services/network => generated 88 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:44.809377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5db4891-6bd4-4cdd-8e17-979a03bf2160 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:44.830084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01e63bc-95ac-4422-95f7-f1f68e34381c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:44.836065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:44.840052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:44.880675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05633c9a-9fe2-489a-8d44-75b4dd7776f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:44.881401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05633c9a-9fe2-489a-8d44-75b4dd7776f6 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:44.881691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05633c9a-9fe2-489a-8d44-75b4dd7776f6 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:44.881969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05633c9a-9fe2-489a-8d44-75b4dd7776f6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:44.893134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05633c9a-9fe2-489a-8d44-75b4dd7776f6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:44.894618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05633c9a-9fe2-489a-8d44-75b4dd7776f6 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-05633c9a-9fe2-489a-8d44-75b4dd7776f6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:44.915601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:44.916591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:44.917581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:44.917581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:44.917581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:44.917581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:44.917581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:44.917581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:44.917581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:44.917581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:44.918029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 353/715] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:44.923751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05633c9a-9fe2-489a-8d44-75b4dd7776f6 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:44.940527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5db4891-6bd4-4cdd-8e17-979a03bf2160 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:44.986201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c05cf41-e2b4-41a3-8fee-e843230080cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:44.986914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c05cf41-e2b4-41a3-8fee-e843230080cf None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:44.987207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c05cf41-e2b4-41a3-8fee-e843230080cf None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:44.987483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c05cf41-e2b4-41a3-8fee-e843230080cf None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:44.988519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c05cf41-e2b4-41a3-8fee-e843230080cf None admin] RBAC: Adding query filter params (name=network) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:44.988882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c05cf41-e2b4-41a3-8fee-e843230080cf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:44.990344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c05cf41-e2b4-41a3-8fee-e843230080cf None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:44.997171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 363/716] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:34:44 2022] GET /identity/v3/services?name=network => generated 124 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:45.003929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c05cf41-e2b4-41a3-8fee-e843230080cf None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:45.043827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05633c9a-9fe2-489a-8d44-75b4dd7776f6 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:45.085180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f91fcb-3807-4574-9529-b55080ebc2ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:45.085810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f91fcb-3807-4574-9529-b55080ebc2ca None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:45.086151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f91fcb-3807-4574-9529-b55080ebc2ca None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:45.086433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f91fcb-3807-4574-9529-b55080ebc2ca None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:45.087199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f91fcb-3807-4574-9529-b55080ebc2ca None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:45.087512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f91fcb-3807-4574-9529-b55080ebc2ca None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:45.089231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f91fcb-3807-4574-9529-b55080ebc2ca None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:45.103684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 354/717] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:45.114447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:45.118985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f91fcb-3807-4574-9529-b55080ebc2ca None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:45.132979 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c05cf41-e2b4-41a3-8fee-e843230080cf None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:45.183168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2ee07a-8754-46f5-8a74-4ea7035301d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:45.183814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2ee07a-8754-46f5-8a74-4ea7035301d9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:45.184064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2ee07a-8754-46f5-8a74-4ea7035301d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:45.184335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2ee07a-8754-46f5-8a74-4ea7035301d9 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:45.185050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2ee07a-8754-46f5-8a74-4ea7035301d9 None admin] RBAC: Adding query filter params (type=network) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:45.185353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2ee07a-8754-46f5-8a74-4ea7035301d9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:45.187238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a2ee07a-8754-46f5-8a74-4ea7035301d9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:45.189244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:45.193230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 364/718] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:34:45 2022] GET /identity/v3/services?type=network => generated 124 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:45.248990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821ff206-d2a0-4a43-b89c-1647a08d6c11 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:45.249330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821ff206-d2a0-4a43-b89c-1647a08d6c11 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:45.249548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821ff206-d2a0-4a43-b89c-1647a08d6c11 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:45.255405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 365/719] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:45 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:45.262996 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6409bcba-632f-4de0-bd8d-69270fd73b2c None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:45.264480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6409bcba-632f-4de0-bd8d-69270fd73b2c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:45.264837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6409bcba-632f-4de0-bd8d-69270fd73b2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:45.296407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6409bcba-632f-4de0-bd8d-69270fd73b2c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:45.296407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6409bcba-632f-4de0-bd8d-69270fd73b2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:45.335086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f91fcb-3807-4574-9529-b55080ebc2ca None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:45.404694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 366/720] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:34:45.431143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6409bcba-632f-4de0-bd8d-69270fd73b2c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:45.465343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f506a05a-d24c-4749-af47-770738c9a6a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:45.466404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f506a05a-d24c-4749-af47-770738c9a6a1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:45.466694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f506a05a-d24c-4749-af47-770738c9a6a1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:45.466975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f506a05a-d24c-4749-af47-770738c9a6a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:45.467794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f506a05a-d24c-4749-af47-770738c9a6a1 None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=441bcfe715d44efea69d556677094857, user.id=58c33a72a871432b88da4f0bf1bc514c) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:45.468115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f506a05a-d24c-4749-af47-770738c9a6a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:45.470243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f506a05a-d24c-4749-af47-770738c9a6a1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:45.477385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 355/721] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:45 2022] GET /identity/v3/role_assignments?user.id=58c33a72a871432b88da4f0bf1bc514c&scope.project.id=441bcfe715d44efea69d556677094857&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 259 bytes in 359 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:45.481958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e93a3fa-0ba8-45e6-be75-57df2e7133f8 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:45.481958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e93a3fa-0ba8-45e6-be75-57df2e7133f8 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:45.481958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e93a3fa-0ba8-45e6-be75-57df2e7133f8 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:45.482571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 356/722] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:45 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:45.488487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c67cb8-21aa-4544-9ce2-89f92591dce4 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:45.488841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c67cb8-21aa-4544-9ce2-89f92591dce4 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:45.489111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c67cb8-21aa-4544-9ce2-89f92591dce4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:45.512248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-b6c67cb8-21aa-4544-9ce2-89f92591dce4 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:45.513940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b6c67cb8-21aa-4544-9ce2-89f92591dce4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:45.554715 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:45.582541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6409bcba-632f-4de0-bd8d-69270fd73b2c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:45.610865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 357/723] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 29 04:34:45.630408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1880da-352c-4a86-956d-95ed81d98efa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:45.631106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1880da-352c-4a86-956d-95ed81d98efa None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:45.631398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1880da-352c-4a86-956d-95ed81d98efa None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:45.631677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1880da-352c-4a86-956d-95ed81d98efa None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:45.632410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1880da-352c-4a86-956d-95ed81d98efa None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:45.633895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1880da-352c-4a86-956d-95ed81d98efa None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:45.661614 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 367/724] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:34:45 2022] POST /identity/v3/services => generated 251 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:34:45.665476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c67cb8-21aa-4544-9ce2-89f92591dce4 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:45.690875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:45.783124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c67cb8-21aa-4544-9ce2-89f92591dce4 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:45.833408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e3e3d1-3d15-4e22-a465-108a94c459d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:45.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e3e3d1-3d15-4e22-a465-108a94c459d1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:45.834353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e3e3d1-3d15-4e22-a465-108a94c459d1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:45.834640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e3e3d1-3d15-4e22-a465-108a94c459d1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:45.847748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e3e3d1-3d15-4e22-a465-108a94c459d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-78e3e3d1-3d15-4e22-a465-108a94c459d1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:45.852815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:45.853754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:45.854571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 358/725] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:45.862505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1880da-352c-4a86-956d-95ed81d98efa None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:45.990202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1880da-352c-4a86-956d-95ed81d98efa None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:46.028033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2e8f3f-4b11-4e64-bffc-472eed5ff7f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:46.028688 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2e8f3f-4b11-4e64-bffc-472eed5ff7f1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:46.028977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2e8f3f-4b11-4e64-bffc-472eed5ff7f1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:46.029265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2e8f3f-4b11-4e64-bffc-472eed5ff7f1 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:46.029991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2e8f3f-4b11-4e64-bffc-472eed5ff7f1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:46.030301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2e8f3f-4b11-4e64-bffc-472eed5ff7f1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:46.031680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2e8f3f-4b11-4e64-bffc-472eed5ff7f1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:46.034605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:46.043473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 368/726] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:45 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:46.049822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e3e3d1-3d15-4e22-a465-108a94c459d1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:46.186602 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:46.227427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e3e3d1-3d15-4e22-a465-108a94c459d1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:46.290891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fd4d6d-6049-40c0-8059-c0f9d7008121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:46.291468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fd4d6d-6049-40c0-8059-c0f9d7008121 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:46.291762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fd4d6d-6049-40c0-8059-c0f9d7008121 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:46.292051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fd4d6d-6049-40c0-8059-c0f9d7008121 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:46.304720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fd4d6d-6049-40c0-8059-c0f9d7008121 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:46.306235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fd4d6d-6049-40c0-8059-c0f9d7008121 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-58fd4d6d-6049-40c0-8059-c0f9d7008121 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:46.321614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:46.322525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:46.323402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:46.323402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:46.323402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:46.323402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:46.323402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 359/727] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:46.328671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2e8f3f-4b11-4e64-bffc-472eed5ff7f1 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:46.488141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2e8f3f-4b11-4e64-bffc-472eed5ff7f1 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:46.537483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4eda78-ec9d-48d4-a80a-84e42932e833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:46.538685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4eda78-ec9d-48d4-a80a-84e42932e833 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:46.539711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4eda78-ec9d-48d4-a80a-84e42932e833 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:46.540830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4eda78-ec9d-48d4-a80a-84e42932e833 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:46.541594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4eda78-ec9d-48d4-a80a-84e42932e833 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:46.542120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4eda78-ec9d-48d4-a80a-84e42932e833 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:46.543491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4eda78-ec9d-48d4-a80a-84e42932e833 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:46.560420 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 369/728] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:46.568063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fd4d6d-6049-40c0-8059-c0f9d7008121 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:46.763413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fd4d6d-6049-40c0-8059-c0f9d7008121 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:46.811913 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:46.834684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d9700-a67b-4f8a-aca4-a6eca3475564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:46.836118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d9700-a67b-4f8a-aca4-a6eca3475564 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:46.836629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d9700-a67b-4f8a-aca4-a6eca3475564 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:46.837133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d9700-a67b-4f8a-aca4-a6eca3475564 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:46.853669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0d9700-a67b-4f8a-aca4-a6eca3475564 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:46.855414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0d9700-a67b-4f8a-aca4-a6eca3475564 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-de0d9700-a67b-4f8a-aca4-a6eca3475564 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:46.879024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:46.880020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:46.880975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:46.880975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:46.880975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:46.880975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:46.880975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:46.880975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:46.880975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:46.880975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:46.880975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:46.880975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:46.883516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 360/729] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:46 2022] GET /identity/v3/users/swift => generated 83 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:46.888886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4eda78-ec9d-48d4-a80a-84e42932e833 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:47.022265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:47.079065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4eda78-ec9d-48d4-a80a-84e42932e833 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:47.123820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae30c26e-977e-44fb-80fc-b1176d4b8b7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:47.124451 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae30c26e-977e-44fb-80fc-b1176d4b8b7b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:47.124766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae30c26e-977e-44fb-80fc-b1176d4b8b7b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:47.125047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae30c26e-977e-44fb-80fc-b1176d4b8b7b None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:47.126480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae30c26e-977e-44fb-80fc-b1176d4b8b7b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:47.126793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae30c26e-977e-44fb-80fc-b1176d4b8b7b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:47.128229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae30c26e-977e-44fb-80fc-b1176d4b8b7b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:47.151465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 370/730] 10.222.0.40 () {56 vars in 1135 bytes} [Fri Jul 29 04:34:46 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:47.157248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d9700-a67b-4f8a-aca4-a6eca3475564 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:47.315870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d9700-a67b-4f8a-aca4-a6eca3475564 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:47.355092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffbaa2e-c867-4a50-b1fa-7b1a0d49df87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:47.355648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffbaa2e-c867-4a50-b1fa-7b1a0d49df87 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:47.355942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffbaa2e-c867-4a50-b1fa-7b1a0d49df87 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:47.356226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffbaa2e-c867-4a50-b1fa-7b1a0d49df87 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:47.450776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:47.463654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffbaa2e-c867-4a50-b1fa-7b1a0d49df87 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:47.465279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffbaa2e-c867-4a50-b1fa-7b1a0d49df87 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:47.613127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48b2b8e-5486-4f6a-b8d4-81b2f7e60fef None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:47.613648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48b2b8e-5486-4f6a-b8d4-81b2f7e60fef None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:47.613648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48b2b8e-5486-4f6a-b8d4-81b2f7e60fef None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:47.616551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 371/731] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:34:47.633529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c976dc8-a4ee-43b5-9d25-a6e2330a7e1b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:47.634172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c976dc8-a4ee-43b5-9d25-a6e2330a7e1b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:47.634682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c976dc8-a4ee-43b5-9d25-a6e2330a7e1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:47.680081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-8c976dc8-a4ee-43b5-9d25-a6e2330a7e1b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:47.681693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8c976dc8-a4ee-43b5-9d25-a6e2330a7e1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:47.702796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 361/732] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:47 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 546 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 29 04:34:47.708781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffbaa2e-c867-4a50-b1fa-7b1a0d49df87 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:47.816807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 372/733] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:47 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:47.840965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffbaa2e-c867-4a50-b1fa-7b1a0d49df87 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:47.854056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c976dc8-a4ee-43b5-9d25-a6e2330a7e1b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:47.865523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:47.911409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad10051-c73c-4030-9ae4-520dbf808fde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:47.912071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad10051-c73c-4030-9ae4-520dbf808fde None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:47.912359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad10051-c73c-4030-9ae4-520dbf808fde None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:47.912658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad10051-c73c-4030-9ae4-520dbf808fde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:47.932645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad10051-c73c-4030-9ae4-520dbf808fde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:47.934126 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad10051-c73c-4030-9ae4-520dbf808fde None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:47.935840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-3ad10051-c73c-4030-9ae4-520dbf808fde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:47.958241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:47.959190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:47.960071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:47.960071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:47.960071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:47.960071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:47.960071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 362/734] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:47.960837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d977869-c901-4fe1-97a6-cf7d29ae9467 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:47.961002 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d977869-c901-4fe1-97a6-cf7d29ae9467 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:47.961276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d977869-c901-4fe1-97a6-cf7d29ae9467 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:47.966375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 363/735] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:47.967163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d977869-c901-4fe1-97a6-cf7d29ae9467 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:48.014484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c976dc8-a4ee-43b5-9d25-a6e2330a7e1b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:48.052074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534d6d6b-906d-4f45-827e-9450b75cd0e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:48.052787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534d6d6b-906d-4f45-827e-9450b75cd0e1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.053069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534d6d6b-906d-4f45-827e-9450b75cd0e1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.053343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534d6d6b-906d-4f45-827e-9450b75cd0e1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.065146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534d6d6b-906d-4f45-827e-9450b75cd0e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-534d6d6b-906d-4f45-827e-9450b75cd0e1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:48.070131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:48.071046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:48.071879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 373/736] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:48.073612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270c6f55-b4e4-4c3a-a5d9-d1439022b4b4 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.073612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270c6f55-b4e4-4c3a-a5d9-d1439022b4b4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.073612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270c6f55-b4e4-4c3a-a5d9-d1439022b4b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.092569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-270c6f55-b4e4-4c3a-a5d9-d1439022b4b4 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:48.093798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-270c6f55-b4e4-4c3a-a5d9-d1439022b4b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:48.119432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d977869-c901-4fe1-97a6-cf7d29ae9467 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:48.168136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33de1791-2e36-4c72-b0bd-13dedba7c35a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:48.169149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33de1791-2e36-4c72-b0bd-13dedba7c35a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.170230 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33de1791-2e36-4c72-b0bd-13dedba7c35a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.170541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33de1791-2e36-4c72-b0bd-13dedba7c35a None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.181424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33de1791-2e36-4c72-b0bd-13dedba7c35a None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:48.181424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33de1791-2e36-4c72-b0bd-13dedba7c35a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:48.181424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33de1791-2e36-4c72-b0bd-13dedba7c35a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:48.188218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 374/737] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:48 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:48.189135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270c6f55-b4e4-4c3a-a5d9-d1439022b4b4 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:48.201714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 364/738] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:48.204080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e213492-367f-4318-8c68-939de95e7504 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.204564 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e213492-367f-4318-8c68-939de95e7504 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.205057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e213492-367f-4318-8c68-939de95e7504 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.208222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 365/739] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:48.209084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e213492-367f-4318-8c68-939de95e7504 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:48.220265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:48.244034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:48.323409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270c6f55-b4e4-4c3a-a5d9-d1439022b4b4 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:48.332671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e213492-367f-4318-8c68-939de95e7504 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:48.360794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bebf0ed8-3476-4ae8-a1f7-30131302df0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:48.361416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebf0ed8-3476-4ae8-a1f7-30131302df0d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.361699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebf0ed8-3476-4ae8-a1f7-30131302df0d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.361976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bebf0ed8-3476-4ae8-a1f7-30131302df0d None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.362691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebf0ed8-3476-4ae8-a1f7-30131302df0d None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:48.362995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebf0ed8-3476-4ae8-a1f7-30131302df0d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:48.364365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bebf0ed8-3476-4ae8-a1f7-30131302df0d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:48.370884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ed3c60-7752-41a2-a520-22b1518d6bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:48.371472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ed3c60-7752-41a2-a520-22b1518d6bca None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.371762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ed3c60-7752-41a2-a520-22b1518d6bca None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.372044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ed3c60-7752-41a2-a520-22b1518d6bca None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.378028 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 375/740] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:48.379309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24e9ae8-523a-4f6e-809b-e1192b9e9841 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.379416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ed3c60-7752-41a2-a520-22b1518d6bca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:48.379599 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24e9ae8-523a-4f6e-809b-e1192b9e9841 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.379877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24e9ae8-523a-4f6e-809b-e1192b9e9841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.380792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ed3c60-7752-41a2-a520-22b1518d6bca None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-e2ed3c60-7752-41a2-a520-22b1518d6bca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:48.393067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:48.393980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:48.394857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:48.394857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:48.394857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:48.394857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:48.394857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:48.394857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:48.394857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:48.394857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:48.394857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 366/741] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:48.395342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ed3c60-7752-41a2-a520-22b1518d6bca None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:48.400110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-b24e9ae8-523a-4f6e-809b-e1192b9e9841 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:48.401559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b24e9ae8-523a-4f6e-809b-e1192b9e9841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:48.460551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:48.481881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 376/742] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:48 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 29 04:34:48.482731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24e9ae8-523a-4f6e-809b-e1192b9e9841 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:48.507213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ed3c60-7752-41a2-a520-22b1518d6bca None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:48.509829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:48.541402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d6511-12bc-464f-b9ee-f5676ee9c43d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:48.542046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d6511-12bc-464f-b9ee-f5676ee9c43d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.542337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d6511-12bc-464f-b9ee-f5676ee9c43d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.542619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d6511-12bc-464f-b9ee-f5676ee9c43d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.552850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512d6511-12bc-464f-b9ee-f5676ee9c43d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-512d6511-12bc-464f-b9ee-f5676ee9c43d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:48.559130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:48.560066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:48.560938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 367/743] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:48.561638 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d6511-12bc-464f-b9ee-f5676ee9c43d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:48.583103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24e9ae8-523a-4f6e-809b-e1192b9e9841 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:48.618246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a9587f1-bb88-4acb-963d-54ab681ceabc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:48.618895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9587f1-bb88-4acb-963d-54ab681ceabc None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.619183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9587f1-bb88-4acb-963d-54ab681ceabc None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.619462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a9587f1-bb88-4acb-963d-54ab681ceabc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.629697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9587f1-bb88-4acb-963d-54ab681ceabc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:48.631114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a9587f1-bb88-4acb-963d-54ab681ceabc None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-5a9587f1-bb88-4acb-963d-54ab681ceabc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:48.646415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:48.647499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:48.648389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:48.648389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:48.648389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:48.648389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:48.648389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 377/744] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:48.652426 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2df1cb-7ff7-4de3-8a95-7cf0a1d1c0a5 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.652742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2df1cb-7ff7-4de3-8a95-7cf0a1d1c0a5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.653564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2df1cb-7ff7-4de3-8a95-7cf0a1d1c0a5 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.654551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 378/745] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:48.655254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2df1cb-7ff7-4de3-8a95-7cf0a1d1c0a5 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:48.667940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d6511-12bc-464f-b9ee-f5676ee9c43d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:48.689749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:48.704209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50f2329-fdf0-4b9a-8a8e-638c60a7083f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:48.704924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f2329-fdf0-4b9a-8a8e-638c60a7083f None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.705208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f2329-fdf0-4b9a-8a8e-638c60a7083f None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50f2329-fdf0-4b9a-8a8e-638c60a7083f None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.706237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50f2329-fdf0-4b9a-8a8e-638c60a7083f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:48.706549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50f2329-fdf0-4b9a-8a8e-638c60a7083f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:48.707972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50f2329-fdf0-4b9a-8a8e-638c60a7083f None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:48.725307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 368/746] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:48.726123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50f2329-fdf0-4b9a-8a8e-638c60a7083f None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:48.746244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:48.767762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2df1cb-7ff7-4de3-8a95-7cf0a1d1c0a5 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:48.802045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecb05dc-8709-41c0-a7d6-88a781a83497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:48.802678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecb05dc-8709-41c0-a7d6-88a781a83497 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.802962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecb05dc-8709-41c0-a7d6-88a781a83497 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.803246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecb05dc-8709-41c0-a7d6-88a781a83497 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.803934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecb05dc-8709-41c0-a7d6-88a781a83497 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:48.805333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecb05dc-8709-41c0-a7d6-88a781a83497 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:48.819919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 379/747] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:34:48 2022] POST /identity/v3/services => generated 240 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:34:48.830667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecb05dc-8709-41c0-a7d6-88a781a83497 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:48.830778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50f2329-fdf0-4b9a-8a8e-638c60a7083f None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:48.878942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d819bb54-50e9-4e82-b26e-9f747e8603e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:48.879647 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d819bb54-50e9-4e82-b26e-9f747e8603e9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.879785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d819bb54-50e9-4e82-b26e-9f747e8603e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.879976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d819bb54-50e9-4e82-b26e-9f747e8603e9 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.880744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d819bb54-50e9-4e82-b26e-9f747e8603e9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:48.881057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d819bb54-50e9-4e82-b26e-9f747e8603e9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:48.882544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d819bb54-50e9-4e82-b26e-9f747e8603e9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:48.898118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 369/748] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:48.898118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ad0fe-093a-4917-bd0b-0d0c04b24622 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.898118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ad0fe-093a-4917-bd0b-0d0c04b24622 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.898118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080ad0fe-093a-4917-bd0b-0d0c04b24622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.917920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-080ad0fe-093a-4917-bd0b-0d0c04b24622 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:48.919222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-080ad0fe-093a-4917-bd0b-0d0c04b24622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:48.952093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecb05dc-8709-41c0-a7d6-88a781a83497 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:48.953591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:48.994360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94df5502-f299-486e-b507-f8b008b0017d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:48.994982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94df5502-f299-486e-b507-f8b008b0017d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:48.995265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94df5502-f299-486e-b507-f8b008b0017d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:48.995546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94df5502-f299-486e-b507-f8b008b0017d None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:48.996631 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94df5502-f299-486e-b507-f8b008b0017d None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:48.996631 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94df5502-f299-486e-b507-f8b008b0017d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:48.998638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94df5502-f299-486e-b507-f8b008b0017d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:49.001237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 370/749] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:48 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:34:49.002113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080ad0fe-093a-4917-bd0b-0d0c04b24622 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:49.015589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:49.019879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 380/750] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:49.022317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94df5502-f299-486e-b507-f8b008b0017d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:49.128084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080ad0fe-093a-4917-bd0b-0d0c04b24622 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:49.132470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94df5502-f299-486e-b507-f8b008b0017d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:49.162498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7591c60e-ae51-463a-bc6f-825de72f4fc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:49.163123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7591c60e-ae51-463a-bc6f-825de72f4fc2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:49.163406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7591c60e-ae51-463a-bc6f-825de72f4fc2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:49.163683 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7591c60e-ae51-463a-bc6f-825de72f4fc2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:49.164470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7591c60e-ae51-463a-bc6f-825de72f4fc2 None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=441bcfe715d44efea69d556677094857, user.id=bbfe92babca048d882cb59cb29b3ed97) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:49.164814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7591c60e-ae51-463a-bc6f-825de72f4fc2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:49.166239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7591c60e-ae51-463a-bc6f-825de72f4fc2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:49.170835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8766745e-549f-486d-b3f4-1266fc033928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:49.171485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8766745e-549f-486d-b3f4-1266fc033928 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:49.181073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8766745e-549f-486d-b3f4-1266fc033928 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:49.181272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 371/751] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/role_assignments?user.id=bbfe92babca048d882cb59cb29b3ed97&scope.project.id=441bcfe715d44efea69d556677094857&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:49.181353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8766745e-549f-486d-b3f4-1266fc033928 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:49.181449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7591c60e-ae51-463a-bc6f-825de72f4fc2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:49.194508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8766745e-549f-486d-b3f4-1266fc033928 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:49.196156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8766745e-549f-486d-b3f4-1266fc033928 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-8766745e-549f-486d-b3f4-1266fc033928 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:49.215992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:49.216993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:49.217853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:49.217853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:49.217853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:49.217853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:49.217853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 381/752] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:49.218752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8766745e-549f-486d-b3f4-1266fc033928 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:49.225220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:49.283806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:49.340933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7591c60e-ae51-463a-bc6f-825de72f4fc2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:49.343626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8766745e-549f-486d-b3f4-1266fc033928 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:49.380683 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3009da-5744-4de4-af54-5ea2f0a5c5d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:49.381704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3009da-5744-4de4-af54-5ea2f0a5c5d4 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:49.383326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3009da-5744-4de4-af54-5ea2f0a5c5d4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:49.383447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fdc32c-325d-4ea3-89c4-13cc8bda9dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:49.383556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3009da-5744-4de4-af54-5ea2f0a5c5d4 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:49.383663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fdc32c-325d-4ea3-89c4-13cc8bda9dd4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:49.384293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fdc32c-325d-4ea3-89c4-13cc8bda9dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:49.384293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fdc32c-325d-4ea3-89c4-13cc8bda9dd4 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:49.384928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fdc32c-325d-4ea3-89c4-13cc8bda9dd4 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:49.387084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fdc32c-325d-4ea3-89c4-13cc8bda9dd4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:49.392415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3009da-5744-4de4-af54-5ea2f0a5c5d4 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:49.393993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a3009da-5744-4de4-af54-5ea2f0a5c5d4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-e3fdc32c-325d-4ea3-89c4-13cc8bda9dd4 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:49.397664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:49.398590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 29 04:34:49.399444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:49.400750 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 382/753] 10.222.0.40 () {56 vars in 1129 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:49.404623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fdc32c-325d-4ea3-89c4-13cc8bda9dd4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-0a3009da-5744-4de4-af54-5ea2f0a5c5d4 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:49.409886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:49.411045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 372/754] 10.222.0.40 () {56 vars in 1117 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:49.412185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3009da-5744-4de4-af54-5ea2f0a5c5d4 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:49.529272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fdc32c-325d-4ea3-89c4-13cc8bda9dd4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:49.535514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:49.546428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a3009da-5744-4de4-af54-5ea2f0a5c5d4 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:49.551308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:49.574519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8759c9-6649-40ee-a4a9-2bb420133bcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:49.575553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8759c9-6649-40ee-a4a9-2bb420133bcd None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:49.575965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8759c9-6649-40ee-a4a9-2bb420133bcd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:49.576338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8759c9-6649-40ee-a4a9-2bb420133bcd None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:49.577318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8759c9-6649-40ee-a4a9-2bb420133bcd None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:49.577727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8759c9-6649-40ee-a4a9-2bb420133bcd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:49.580516 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8759c9-6649-40ee-a4a9-2bb420133bcd None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:49.587514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491dc7b2-1415-4709-b701-1d7fd96a2255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:49.588378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491dc7b2-1415-4709-b701-1d7fd96a2255 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:49.588901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491dc7b2-1415-4709-b701-1d7fd96a2255 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:49.589327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491dc7b2-1415-4709-b701-1d7fd96a2255 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:49.590262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491dc7b2-1415-4709-b701-1d7fd96a2255 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:49.590718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491dc7b2-1415-4709-b701-1d7fd96a2255 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:49.592338 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491dc7b2-1415-4709-b701-1d7fd96a2255 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:49.596227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 383/755] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:49.597269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8759c9-6649-40ee-a4a9-2bb420133bcd None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:49.602519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 373/756] 10.222.0.40 () {56 vars in 1124 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:49.603775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491dc7b2-1415-4709-b701-1d7fd96a2255 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:49.725061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8759c9-6649-40ee-a4a9-2bb420133bcd None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:49.737878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491dc7b2-1415-4709-b701-1d7fd96a2255 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:49.766501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6474995-ca8e-433f-847e-78ba513c9cee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:49.767412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6474995-ca8e-433f-847e-78ba513c9cee None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:49.767939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6474995-ca8e-433f-847e-78ba513c9cee None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:49.768464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6474995-ca8e-433f-847e-78ba513c9cee None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:49.769521 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6474995-ca8e-433f-847e-78ba513c9cee None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:49.770044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6474995-ca8e-433f-847e-78ba513c9cee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:49.771675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6474995-ca8e-433f-847e-78ba513c9cee None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:49.780352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a42e61-242f-4756-b2f6-f6be39f25bb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:49.781023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a42e61-242f-4756-b2f6-f6be39f25bb3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:49.781308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a42e61-242f-4756-b2f6-f6be39f25bb3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:49.782235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a42e61-242f-4756-b2f6-f6be39f25bb3 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:49.792005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a42e61-242f-4756-b2f6-f6be39f25bb3 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:49.797917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a42e61-242f-4756-b2f6-f6be39f25bb3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:49.798777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 384/757] 10.222.0.40 () {56 vars in 1149 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:49.799875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6474995-ca8e-433f-847e-78ba513c9cee None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-d5a42e61-242f-4756-b2f6-f6be39f25bb3 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:49.807490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:49.808425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:49.809433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 374/758] 10.222.0.40 () {56 vars in 1093 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:49.810021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a42e61-242f-4756-b2f6-f6be39f25bb3 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:49.834620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:49.841425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:49.932325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6474995-ca8e-433f-847e-78ba513c9cee None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:49.933947 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a42e61-242f-4756-b2f6-f6be39f25bb3 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:49.972537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10a4f3f-4a18-4b89-81d8-e8b706da8091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:49.973628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10a4f3f-4a18-4b89-81d8-e8b706da8091 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:49.974121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10a4f3f-4a18-4b89-81d8-e8b706da8091 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:49.974608 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10a4f3f-4a18-4b89-81d8-e8b706da8091 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:49.975552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10a4f3f-4a18-4b89-81d8-e8b706da8091 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:49.976104 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10a4f3f-4a18-4b89-81d8-e8b706da8091 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:49.977803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10a4f3f-4a18-4b89-81d8-e8b706da8091 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:49.979884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4d3696-92ab-46c9-999b-8aaa4f361c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:49.981076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4d3696-92ab-46c9-999b-8aaa4f361c27 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:49.981423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4d3696-92ab-46c9-999b-8aaa4f361c27 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:49.981709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4d3696-92ab-46c9-999b-8aaa4f361c27 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:49.988929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 385/759] 10.222.0.40 () {56 vars in 1124 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:49.990072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10a4f3f-4a18-4b89-81d8-e8b706da8091 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:50.076879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4d3696-92ab-46c9-999b-8aaa4f361c27 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:50.079451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4d3696-92ab-46c9-999b-8aaa4f361c27 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:50.100316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:50.104439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:50.106133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10a4f3f-4a18-4b89-81d8-e8b706da8091 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:50.144236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd204cb5-745f-497c-a449-1d53d955afad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:50.145252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd204cb5-745f-497c-a449-1d53d955afad None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:50.145745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd204cb5-745f-497c-a449-1d53d955afad None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:50.146226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd204cb5-745f-497c-a449-1d53d955afad None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:50.147243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd204cb5-745f-497c-a449-1d53d955afad None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:50.147772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd204cb5-745f-497c-a449-1d53d955afad None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:50.149437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd204cb5-745f-497c-a449-1d53d955afad None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:50.168792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 386/760] 10.222.0.40 () {56 vars in 1133 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:50.169859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd204cb5-745f-497c-a449-1d53d955afad None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:50.220188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 375/761] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:49 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:34:50.221429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4d3696-92ab-46c9-999b-8aaa4f361c27 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:50.295072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd204cb5-745f-497c-a449-1d53d955afad None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:50.327540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4d3696-92ab-46c9-999b-8aaa4f361c27 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:50.342213 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd59b76-57f1-4c7f-943e-6817f1649cf6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:50.343121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd59b76-57f1-4c7f-943e-6817f1649cf6 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:50.343408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd59b76-57f1-4c7f-943e-6817f1649cf6 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:50.344405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cd59b76-57f1-4c7f-943e-6817f1649cf6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:50.345693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd59b76-57f1-4c7f-943e-6817f1649cf6 None admin] RBAC: Adding query filter params (interface=public, service_id=90ce945c32814729ad82138f7dd771dc, region_id=RegionOne) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:50.346012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd59b76-57f1-4c7f-943e-6817f1649cf6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:50.347064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:50.349191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cd59b76-57f1-4c7f-943e-6817f1649cf6 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:50.362392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 387/762] 10.222.0.40 () {56 vars in 1251 bytes} [Fri Jul 29 04:34:50 2022] GET /identity/v3/endpoints?service_id=90ce945c32814729ad82138f7dd771dc&interface=public®ion_id=RegionOne => generated 194 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:50.363768 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd59b76-57f1-4c7f-943e-6817f1649cf6 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:50.368450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a14ca70-0987-470a-8ea3-76d6bca032e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:50.369223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14ca70-0987-470a-8ea3-76d6bca032e6 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:50.369587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14ca70-0987-470a-8ea3-76d6bca032e6 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:50.369944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a14ca70-0987-470a-8ea3-76d6bca032e6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:50.381795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a14ca70-0987-470a-8ea3-76d6bca032e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:50.383433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a14ca70-0987-470a-8ea3-76d6bca032e6 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:50.391774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-8a14ca70-0987-470a-8ea3-76d6bca032e6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:50.400157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:50.401083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:50.401974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:50.401974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:50.401974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:50.401974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:50.406037 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 376/763] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:50.406997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a14ca70-0987-470a-8ea3-76d6bca032e6 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:50.485848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cd59b76-57f1-4c7f-943e-6817f1649cf6 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:50.523706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53be8bc-14ac-4c1a-a3c7-9788bb95d449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:50.524701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53be8bc-14ac-4c1a-a3c7-9788bb95d449 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:50.525174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53be8bc-14ac-4c1a-a3c7-9788bb95d449 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:50.525659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53be8bc-14ac-4c1a-a3c7-9788bb95d449 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:50.530398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a14ca70-0987-470a-8ea3-76d6bca032e6 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:50.537761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53be8bc-14ac-4c1a-a3c7-9788bb95d449 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:50.539599 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53be8bc-14ac-4c1a-a3c7-9788bb95d449 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c53be8bc-14ac-4c1a-a3c7-9788bb95d449 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:50.557014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:50.557965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:50.558842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:50.558842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:50.558842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:50.558842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:50.560969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 388/764] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:50.562017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53be8bc-14ac-4c1a-a3c7-9788bb95d449 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:50.571641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e6b862-67c9-4344-a385-7e4226d42f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:50.572268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e6b862-67c9-4344-a385-7e4226d42f00 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:50.572552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e6b862-67c9-4344-a385-7e4226d42f00 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:50.573375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e6b862-67c9-4344-a385-7e4226d42f00 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:50.574042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e6b862-67c9-4344-a385-7e4226d42f00 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:50.575869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e6b862-67c9-4344-a385-7e4226d42f00 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:50.583691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 377/765] 10.222.0.40 () {56 vars in 1087 bytes} [Fri Jul 29 04:34:50 2022] GET /identity/v3/services => generated 772 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:50.584718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e6b862-67c9-4344-a385-7e4226d42f00 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:50.672670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:50.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:50.718436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53be8bc-14ac-4c1a-a3c7-9788bb95d449 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:50.771195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373f719d-494b-49b5-bc36-1edd2be5f088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:50.772956 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373f719d-494b-49b5-bc36-1edd2be5f088 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:50.773446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373f719d-494b-49b5-bc36-1edd2be5f088 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:50.775190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373f719d-494b-49b5-bc36-1edd2be5f088 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:50.776057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373f719d-494b-49b5-bc36-1edd2be5f088 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:50.777931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373f719d-494b-49b5-bc36-1edd2be5f088 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:50.779472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373f719d-494b-49b5-bc36-1edd2be5f088 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:50.803464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 389/766] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:50.812945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373f719d-494b-49b5-bc36-1edd2be5f088 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:50.842970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e6b862-67c9-4344-a385-7e4226d42f00 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:50.881748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d9199f-5a38-422e-ab8c-81a608e9ede8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:50.882647 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d9199f-5a38-422e-ab8c-81a608e9ede8 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:50.883088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d9199f-5a38-422e-ab8c-81a608e9ede8 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:50.883508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d9199f-5a38-422e-ab8c-81a608e9ede8 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:50.884472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d9199f-5a38-422e-ab8c-81a608e9ede8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:50.885032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d9199f-5a38-422e-ab8c-81a608e9ede8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:50.886558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d9199f-5a38-422e-ab8c-81a608e9ede8 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:50.900763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 378/767] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:50.902130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ccd55f-9a6c-4e99-8d8d-5e993392f8b3 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:50.902556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ccd55f-9a6c-4e99-8d8d-5e993392f8b3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:50.902836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ccd55f-9a6c-4e99-8d8d-5e993392f8b3 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:50.904594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 379/768] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:50.909501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ccd55f-9a6c-4e99-8d8d-5e993392f8b3 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:50.942038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373f719d-494b-49b5-bc36-1edd2be5f088 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:50.973973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:50.980065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d6602c-b18d-4baf-8056-97efebe18a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:50.980804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d6602c-b18d-4baf-8056-97efebe18a8a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:50.981102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d6602c-b18d-4baf-8056-97efebe18a8a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:50.981400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d6602c-b18d-4baf-8056-97efebe18a8a None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:50.989284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d6602c-b18d-4baf-8056-97efebe18a8a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:50.990811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d6602c-b18d-4baf-8056-97efebe18a8a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-09d6602c-b18d-4baf-8056-97efebe18a8a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:51.006085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:51.007180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:51.008266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:51.008266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:51.008266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:51.008266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:51.008266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:51.008266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:51.008266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:51.008266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:51.010378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 390/769] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:51.011834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6542c741-1c53-4e96-bd65-6f1474e966b7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.012218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6542c741-1c53-4e96-bd65-6f1474e966b7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.012595 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6542c741-1c53-4e96-bd65-6f1474e966b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.029557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:51.032355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6542c741-1c53-4e96-bd65-6f1474e966b7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:51.034174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6542c741-1c53-4e96-bd65-6f1474e966b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:51.043249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ccd55f-9a6c-4e99-8d8d-5e993392f8b3 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:51.088307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e17449-24bb-46c0-95fd-0fb65ebd48b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:51.089589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e17449-24bb-46c0-95fd-0fb65ebd48b5 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.090025 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e17449-24bb-46c0-95fd-0fb65ebd48b5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.090476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e17449-24bb-46c0-95fd-0fb65ebd48b5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.099374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e17449-24bb-46c0-95fd-0fb65ebd48b5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:51.100871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e17449-24bb-46c0-95fd-0fb65ebd48b5 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:51.111892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 391/770] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:34:51.112966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6542c741-1c53-4e96-bd65-6f1474e966b7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-f5e17449-24bb-46c0-95fd-0fb65ebd48b5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:51.123740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:51.124876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:51.126115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:51.126115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:51.126115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:51.126115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:51.126115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:51.126115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:51.126115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:51.126115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:51.127798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 380/771] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:51.133122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e17449-24bb-46c0-95fd-0fb65ebd48b5 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:51.244711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6542c741-1c53-4e96-bd65-6f1474e966b7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:51.258966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e17449-24bb-46c0-95fd-0fb65ebd48b5 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:51.270761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:51.285545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f04f53-92df-4fb3-98a1-ad56261cee3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:51.286257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f04f53-92df-4fb3-98a1-ad56261cee3d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.286635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f04f53-92df-4fb3-98a1-ad56261cee3d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.287009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f04f53-92df-4fb3-98a1-ad56261cee3d None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.287853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f04f53-92df-4fb3-98a1-ad56261cee3d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:51.288310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f04f53-92df-4fb3-98a1-ad56261cee3d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:51.290041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f04f53-92df-4fb3-98a1-ad56261cee3d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:51.299969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a7f8f-7d8d-4e6d-a165-2dc6f1c0a958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:51.300828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a7f8f-7d8d-4e6d-a165-2dc6f1c0a958 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.301258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a7f8f-7d8d-4e6d-a165-2dc6f1c0a958 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.301705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a7f8f-7d8d-4e6d-a165-2dc6f1c0a958 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.302620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a7f8f-7d8d-4e6d-a165-2dc6f1c0a958 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:51.303080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a7f8f-7d8d-4e6d-a165-2dc6f1c0a958 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:51.304666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a7f8f-7d8d-4e6d-a165-2dc6f1c0a958 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:51.304877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 392/772] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:51.305863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f04f53-92df-4fb3-98a1-ad56261cee3d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:51.320513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 381/773] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:51.321565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a7f8f-7d8d-4e6d-a165-2dc6f1c0a958 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:51.337275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:51.415059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f04f53-92df-4fb3-98a1-ad56261cee3d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:51.435265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a7f8f-7d8d-4e6d-a165-2dc6f1c0a958 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:51.452444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5db730-127b-4231-8d67-d7b73afc70bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:51.453258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5db730-127b-4231-8d67-d7b73afc70bd None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.454091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5db730-127b-4231-8d67-d7b73afc70bd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.454606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5db730-127b-4231-8d67-d7b73afc70bd None admin] PATH_INFO: `/v3/services/network` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.455243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5db730-127b-4231-8d67-d7b73afc70bd None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:51.456631 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5db730-127b-4231-8d67-d7b73afc70bd None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-8a5db730-127b-4231-8d67-d7b73afc70bd None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:51.465687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:51.466652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:51.467490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 393/774] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:51 2022] GET /identity/v3/services/network => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:51.468260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5db730-127b-4231-8d67-d7b73afc70bd None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:51.479342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c37a855-2bb3-49a6-b500-9b8e5ff0368f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:51.480019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c37a855-2bb3-49a6-b500-9b8e5ff0368f None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.480302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c37a855-2bb3-49a6-b500-9b8e5ff0368f None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.480582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c37a855-2bb3-49a6-b500-9b8e5ff0368f None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/58c33a72a871432b88da4f0bf1bc514c/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.482544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c37a855-2bb3-49a6-b500-9b8e5ff0368f None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=58c33a72a871432b88da4f0bf1bc514c, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:51.483969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c37a855-2bb3-49a6-b500-9b8e5ff0368f None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:51.497758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 382/775] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:34:51 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/58c33a72a871432b88da4f0bf1bc514c/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:51.499170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb31163-53e1-4ced-925a-414acff53765 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.499459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb31163-53e1-4ced-925a-414acff53765 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.499738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb31163-53e1-4ced-925a-414acff53765 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.501389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 383/776] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:51.506226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb31163-53e1-4ced-925a-414acff53765 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:51.541106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:51.602873 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:51.612191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5db730-127b-4231-8d67-d7b73afc70bd None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:51.645227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb31163-53e1-4ced-925a-414acff53765 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:51.677251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3563b95d-05c3-451c-be08-b45cecc74896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:51.678030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563b95d-05c3-451c-be08-b45cecc74896 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.678394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563b95d-05c3-451c-be08-b45cecc74896 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.678678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3563b95d-05c3-451c-be08-b45cecc74896 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.679482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3563b95d-05c3-451c-be08-b45cecc74896 None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=3f53019d35f244e4959d8340ba6f7e45) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:51.679898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3563b95d-05c3-451c-be08-b45cecc74896 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:51.683074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3563b95d-05c3-451c-be08-b45cecc74896 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:51.690024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 394/777] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:51 2022] GET /identity/v3/role_assignments?user.id=3f53019d35f244e4959d8340ba6f7e45&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 259 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:51.690173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f252271-7031-4703-a586-976535b4e4f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:51.690596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f252271-7031-4703-a586-976535b4e4f9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.690883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f252271-7031-4703-a586-976535b4e4f9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.691160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f252271-7031-4703-a586-976535b4e4f9 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.691849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f252271-7031-4703-a586-976535b4e4f9 None admin] RBAC: Adding query filter params (name=network) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:51.691954 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f31b94e-a311-4a21-82b3-5942d3108bed None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.692191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f31b94e-a311-4a21-82b3-5942d3108bed None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.692289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f252271-7031-4703-a586-976535b4e4f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:51.692561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f31b94e-a311-4a21-82b3-5942d3108bed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.693565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f252271-7031-4703-a586-976535b4e4f9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:51.698898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 384/778] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:34:51 2022] GET /identity/v3/services?name=network => generated 124 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:51.707049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f252271-7031-4703-a586-976535b4e4f9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:51.717457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-9f31b94e-a311-4a21-82b3-5942d3108bed None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:51.718714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9f31b94e-a311-4a21-82b3-5942d3108bed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:51.869240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f252271-7031-4703-a586-976535b4e4f9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:51.870991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 395/779] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:34:51.927981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03878242-7b20-4e13-ac2a-4c32a65ee9a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:51.928421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f31b94e-a311-4a21-82b3-5942d3108bed None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:51.929199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03878242-7b20-4e13-ac2a-4c32a65ee9a2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:51.929777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03878242-7b20-4e13-ac2a-4c32a65ee9a2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:51.930256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03878242-7b20-4e13-ac2a-4c32a65ee9a2 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:51.931145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03878242-7b20-4e13-ac2a-4c32a65ee9a2 None admin] RBAC: Adding query filter params (type=network) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:51.931647 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03878242-7b20-4e13-ac2a-4c32a65ee9a2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:51.933332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03878242-7b20-4e13-ac2a-4c32a65ee9a2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:51.939498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:51.947639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 385/780] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:34:51 2022] GET /identity/v3/services?type=network => generated 350 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:51.952713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03878242-7b20-4e13-ac2a-4c32a65ee9a2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:51.982641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:52.101365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f31b94e-a311-4a21-82b3-5942d3108bed None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:52.116978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03878242-7b20-4e13-ac2a-4c32a65ee9a2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:52.168103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc97de5-00e7-4b47-97e9-5d98a9a7c02a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:52.169030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc97de5-00e7-4b47-97e9-5d98a9a7c02a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:52.169916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc97de5-00e7-4b47-97e9-5d98a9a7c02a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:52.170695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc97de5-00e7-4b47-97e9-5d98a9a7c02a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:52.175780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1028666-b8c2-4bf1-a517-bfc60a81d167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:52.177538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1028666-b8c2-4bf1-a517-bfc60a81d167 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:52.180594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1028666-b8c2-4bf1-a517-bfc60a81d167 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:52.181053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1028666-b8c2-4bf1-a517-bfc60a81d167 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:52.181973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1028666-b8c2-4bf1-a517-bfc60a81d167 None admin] RBAC: Adding query filter params (interface=public, service_id=c6daa391e3b440a5a3c522d4cd85b086, region_id=RegionOne) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:52.182462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1028666-b8c2-4bf1-a517-bfc60a81d167 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:52.183996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1028666-b8c2-4bf1-a517-bfc60a81d167 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:52.195059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc97de5-00e7-4b47-97e9-5d98a9a7c02a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-4fc97de5-00e7-4b47-97e9-5d98a9a7c02a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:52.201364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:52.202497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:52.204267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:52.210186 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 396/782] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:52.210433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 386/782] 10.222.0.40 () {56 vars in 1251 bytes} [Fri Jul 29 04:34:51 2022] GET /identity/v3/endpoints?service_id=c6daa391e3b440a5a3c522d4cd85b086&interface=public®ion_id=RegionOne => generated 194 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:52.218116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc97de5-00e7-4b47-97e9-5d98a9a7c02a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:52.218459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1028666-b8c2-4bf1-a517-bfc60a81d167 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:52.329632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:52.405200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1028666-b8c2-4bf1-a517-bfc60a81d167 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:52.418254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:52.423468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc97de5-00e7-4b47-97e9-5d98a9a7c02a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:52.469631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504655f5-f77f-43b9-8eed-a81cbfd1f7b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:52.470260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504655f5-f77f-43b9-8eed-a81cbfd1f7b9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:52.470544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504655f5-f77f-43b9-8eed-a81cbfd1f7b9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:52.470810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504655f5-f77f-43b9-8eed-a81cbfd1f7b9 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:52.471467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504655f5-f77f-43b9-8eed-a81cbfd1f7b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:52.477150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504655f5-f77f-43b9-8eed-a81cbfd1f7b9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:52.489626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 387/783] 10.222.0.40 () {56 vars in 1087 bytes} [Fri Jul 29 04:34:52 2022] GET /identity/v3/services => generated 772 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:52.492150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd15806-8c71-47d0-9efc-b833b684058c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:52.492946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd15806-8c71-47d0-9efc-b833b684058c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:52.493232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd15806-8c71-47d0-9efc-b833b684058c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:52.493509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd15806-8c71-47d0-9efc-b833b684058c None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:52.494252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd15806-8c71-47d0-9efc-b833b684058c None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:52.494557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd15806-8c71-47d0-9efc-b833b684058c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:52.495972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd15806-8c71-47d0-9efc-b833b684058c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:52.506944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 397/784] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:52.512705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504655f5-f77f-43b9-8eed-a81cbfd1f7b9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:52.668704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504655f5-f77f-43b9-8eed-a81cbfd1f7b9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:52.705997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:52.713272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc60fc11-c3b5-4f15-a698-edf882ae0dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:52.713966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc60fc11-c3b5-4f15-a698-edf882ae0dde None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:52.714259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc60fc11-c3b5-4f15-a698-edf882ae0dde None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:52.714538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc60fc11-c3b5-4f15-a698-edf882ae0dde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:52.728353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc60fc11-c3b5-4f15-a698-edf882ae0dde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:52.730152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc60fc11-c3b5-4f15-a698-edf882ae0dde None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-fc60fc11-c3b5-4f15-a698-edf882ae0dde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:52.747350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:52.748275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:52.749198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:52.749198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:52.749198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:52.749198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:52.755140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 388/785] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:52.758403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd15806-8c71-47d0-9efc-b833b684058c None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:52.870993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd15806-8c71-47d0-9efc-b833b684058c None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:52.879644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b123d-654b-4a75-9152-e382553c28f5 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:52.880227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b123d-654b-4a75-9152-e382553c28f5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:52.880743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2b123d-654b-4a75-9152-e382553c28f5 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:52.886209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 389/786] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:52.887849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1db13-fa0c-4e90-98f6-988365fef70d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:52.888360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1db13-fa0c-4e90-98f6-988365fef70d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:52.888912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1db13-fa0c-4e90-98f6-988365fef70d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:52.908314 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-fca1db13-fa0c-4e90-98f6-988365fef70d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:52.909909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fca1db13-fa0c-4e90-98f6-988365fef70d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:52.920020 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c37fcb-3bc4-4bd6-b07d-40419f99a164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:52.920970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c37fcb-3bc4-4bd6-b07d-40419f99a164 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:52.921465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c37fcb-3bc4-4bd6-b07d-40419f99a164 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:52.921947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c37fcb-3bc4-4bd6-b07d-40419f99a164 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:52.924125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c37fcb-3bc4-4bd6-b07d-40419f99a164 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:52.926435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c37fcb-3bc4-4bd6-b07d-40419f99a164 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:52.928111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c37fcb-3bc4-4bd6-b07d-40419f99a164 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:52.952227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 398/787] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:52.962813 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c37fcb-3bc4-4bd6-b07d-40419f99a164 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:53.008980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:53.055785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 390/788] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:34:53.116546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca1db13-fa0c-4e90-98f6-988365fef70d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:53.135584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c37fcb-3bc4-4bd6-b07d-40419f99a164 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:53.196982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e3ad91-4db4-49ac-ac70-daa573872c12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:53.197569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e3ad91-4db4-49ac-ac70-daa573872c12 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:53.197859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e3ad91-4db4-49ac-ac70-daa573872c12 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:53.198146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e3ad91-4db4-49ac-ac70-daa573872c12 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:53.210214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e3ad91-4db4-49ac-ac70-daa573872c12 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:53.211676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e3ad91-4db4-49ac-ac70-daa573872c12 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-e7e3ad91-4db4-49ac-ac70-daa573872c12 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:53.238245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:53.239251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:53.240164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:53.240164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:53.240164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:53.240164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:53.240164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:53.240164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:53.240164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:53.240164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:53.240164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:53.240164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:53.248062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 399/789] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:52 2022] GET /identity/v3/users/swift => generated 83 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:53.251018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e3ad91-4db4-49ac-ac70-daa573872c12 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:53.319756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca1db13-fa0c-4e90-98f6-988365fef70d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:53.353052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:53.389800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7568d8-dd08-4937-a7a6-3758be99584d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:53.390462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7568d8-dd08-4937-a7a6-3758be99584d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:53.390801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7568d8-dd08-4937-a7a6-3758be99584d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:53.391090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7568d8-dd08-4937-a7a6-3758be99584d None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:53.391730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7568d8-dd08-4937-a7a6-3758be99584d None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:53.393152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7568d8-dd08-4937-a7a6-3758be99584d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-7a7568d8-dd08-4937-a7a6-3758be99584d None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:53.420973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:53.422139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 29 04:34:53.423022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:53.424074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:53.426196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 391/790] 10.222.0.40 () {56 vars in 1129 bytes} [Fri Jul 29 04:34:53 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 310 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:34:53.431551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7568d8-dd08-4937-a7a6-3758be99584d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:53.444633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e3ad91-4db4-49ac-ac70-daa573872c12 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:53.482079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062f89f-adc8-401d-9d5d-272417616aa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:53.482721 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062f89f-adc8-401d-9d5d-272417616aa2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:53.483015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062f89f-adc8-401d-9d5d-272417616aa2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:53.483302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062f89f-adc8-401d-9d5d-272417616aa2 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:53.484211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b062f89f-adc8-401d-9d5d-272417616aa2 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:53.484567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b062f89f-adc8-401d-9d5d-272417616aa2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:53.486084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b062f89f-adc8-401d-9d5d-272417616aa2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:53.510225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 400/791] 10.222.0.40 () {56 vars in 1135 bytes} [Fri Jul 29 04:34:53 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:53.511476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062f89f-adc8-401d-9d5d-272417616aa2 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:53.617518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062f89f-adc8-401d-9d5d-272417616aa2 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:53.638284 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7568d8-dd08-4937-a7a6-3758be99584d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:53.659164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f8f4d4-9d7d-4c25-bf8d-6eec7a810abe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:53.659826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f8f4d4-9d7d-4c25-bf8d-6eec7a810abe None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:53.660118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f8f4d4-9d7d-4c25-bf8d-6eec7a810abe None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:53.665760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f8f4d4-9d7d-4c25-bf8d-6eec7a810abe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:53.677630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f8f4d4-9d7d-4c25-bf8d-6eec7a810abe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:53.679106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f8f4d4-9d7d-4c25-bf8d-6eec7a810abe None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:53.679798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-13f8f4d4-9d7d-4c25-bf8d-6eec7a810abe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:53.695464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:53.696389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:53.697287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:53.697287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:53.697287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:53.697287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:53.701193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 401/792] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:53.702751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f8f4d4-9d7d-4c25-bf8d-6eec7a810abe None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:53.714129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05484bfd-2fcd-4122-a241-f394ca973a7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:53.715183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05484bfd-2fcd-4122-a241-f394ca973a7a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:53.715697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05484bfd-2fcd-4122-a241-f394ca973a7a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:53.716175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05484bfd-2fcd-4122-a241-f394ca973a7a None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:53.717197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05484bfd-2fcd-4122-a241-f394ca973a7a None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:53.717710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05484bfd-2fcd-4122-a241-f394ca973a7a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:53.719971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05484bfd-2fcd-4122-a241-f394ca973a7a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:53.733449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 392/793] 10.222.0.40 () {56 vars in 1124 bytes} [Fri Jul 29 04:34:53 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:53.738644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05484bfd-2fcd-4122-a241-f394ca973a7a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:53.811643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:53.812527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f8f4d4-9d7d-4c25-bf8d-6eec7a810abe None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:53.849638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7896687-53a0-4342-b977-37e0c5cf2020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:53.850266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7896687-53a0-4342-b977-37e0c5cf2020 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:53.852832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7896687-53a0-4342-b977-37e0c5cf2020 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:53.852832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7896687-53a0-4342-b977-37e0c5cf2020 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:53.852832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7896687-53a0-4342-b977-37e0c5cf2020 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:53.852832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7896687-53a0-4342-b977-37e0c5cf2020 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:53.853339 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7896687-53a0-4342-b977-37e0c5cf2020 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:53.866245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 402/794] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:53.869871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9878ec46-a5a5-470a-a2db-cc4c88a5bb42 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:53.869871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9878ec46-a5a5-470a-a2db-cc4c88a5bb42 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:53.869871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9878ec46-a5a5-470a-a2db-cc4c88a5bb42 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:53.870671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 403/795] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:34:53.871831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9878ec46-a5a5-470a-a2db-cc4c88a5bb42 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:53.934234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:53.940143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05484bfd-2fcd-4122-a241-f394ca973a7a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:53.984051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fedf859-28e2-42b9-a906-884e7e2b5b90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:53.985222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fedf859-28e2-42b9-a906-884e7e2b5b90 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:53.985710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fedf859-28e2-42b9-a906-884e7e2b5b90 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:53.986098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fedf859-28e2-42b9-a906-884e7e2b5b90 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:53.987583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fedf859-28e2-42b9-a906-884e7e2b5b90 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:53.987912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fedf859-28e2-42b9-a906-884e7e2b5b90 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:53.989572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fedf859-28e2-42b9-a906-884e7e2b5b90 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:53.998674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9878ec46-a5a5-470a-a2db-cc4c88a5bb42 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:54.002469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 393/796] 10.222.0.40 () {56 vars in 1124 bytes} [Fri Jul 29 04:34:53 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:54.003541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cd0633-7102-4070-bc5d-303334b60818 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.003852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cd0633-7102-4070-bc5d-303334b60818 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.004136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69cd0633-7102-4070-bc5d-303334b60818 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.022933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-69cd0633-7102-4070-bc5d-303334b60818 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:54.024156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-69cd0633-7102-4070-bc5d-303334b60818 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:54.045740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215721d2-0f2a-4916-a152-e6508e46a5aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:54.046756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215721d2-0f2a-4916-a152-e6508e46a5aa None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.048548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215721d2-0f2a-4916-a152-e6508e46a5aa None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.049081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215721d2-0f2a-4916-a152-e6508e46a5aa None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.059534 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215721d2-0f2a-4916-a152-e6508e46a5aa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:54.061507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215721d2-0f2a-4916-a152-e6508e46a5aa None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-215721d2-0f2a-4916-a152-e6508e46a5aa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:54.076774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:54.077765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:54.078670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:54.078670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:54.078670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:54.078670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:54.078670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:54.078670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:54.078670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:54.078670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:54.080094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 404/797] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:54.080966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215721d2-0f2a-4916-a152-e6508e46a5aa None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:54.098265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 394/798] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:34:54.099782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924923f4-88ab-48ff-a347-0faf15079da7 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.100242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924923f4-88ab-48ff-a347-0faf15079da7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.100755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924923f4-88ab-48ff-a347-0faf15079da7 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.102419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 395/799] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:54.103283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924923f4-88ab-48ff-a347-0faf15079da7 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:54.140266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:54.194819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215721d2-0f2a-4916-a152-e6508e46a5aa None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:54.208575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:54.222548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924923f4-88ab-48ff-a347-0faf15079da7 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:54.231554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0224844d-c0d7-4307-b4de-85b274614f4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:54.232242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0224844d-c0d7-4307-b4de-85b274614f4b None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.232461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0224844d-c0d7-4307-b4de-85b274614f4b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.232968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0224844d-c0d7-4307-b4de-85b274614f4b None admin] PATH_INFO: `/v3/endpoints` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.233602 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0224844d-c0d7-4307-b4de-85b274614f4b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:54.235079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0224844d-c0d7-4307-b4de-85b274614f4b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:54.257738 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 405/800] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:34:54 2022] POST /identity/v3/endpoints => generated 346 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:34:54.258812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ee0ef6-c5fb-4fa1-a277-750fc96b25a7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.259098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ee0ef6-c5fb-4fa1-a277-750fc96b25a7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.259377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ee0ef6-c5fb-4fa1-a277-750fc96b25a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.279177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e6ee0ef6-c5fb-4fa1-a277-750fc96b25a7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:54.280399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e6ee0ef6-c5fb-4fa1-a277-750fc96b25a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:54.281299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde490cf-d03f-4d93-a6a8-629c4faf0e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:54.283665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde490cf-d03f-4d93-a6a8-629c4faf0e16 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.285070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde490cf-d03f-4d93-a6a8-629c4faf0e16 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.285190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde490cf-d03f-4d93-a6a8-629c4faf0e16 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.285983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde490cf-d03f-4d93-a6a8-629c4faf0e16 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:54.286294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde490cf-d03f-4d93-a6a8-629c4faf0e16 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:54.287770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde490cf-d03f-4d93-a6a8-629c4faf0e16 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:54.305965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 396/801] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:54.306997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde490cf-d03f-4d93-a6a8-629c4faf0e16 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:54.366090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 406/802] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:34:54.367362 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ee0ef6-c5fb-4fa1-a277-750fc96b25a7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:54.404259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:54.419342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde490cf-d03f-4d93-a6a8-629c4faf0e16 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:54.458687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2db9a0-514e-4cbe-a12b-429880ae3db0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:54.459319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2db9a0-514e-4cbe-a12b-429880ae3db0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.459640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2db9a0-514e-4cbe-a12b-429880ae3db0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.459926 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2db9a0-514e-4cbe-a12b-429880ae3db0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.479973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2db9a0-514e-4cbe-a12b-429880ae3db0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:54.484642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-dd2db9a0-514e-4cbe-a12b-429880ae3db0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:54.486347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:54.487706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 397/803] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:54.488524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2db9a0-514e-4cbe-a12b-429880ae3db0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:54.494386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ee0ef6-c5fb-4fa1-a277-750fc96b25a7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:54.541392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277ad538-5c73-46dd-8ef1-6a616a2f80c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:54.542090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ad538-5c73-46dd-8ef1-6a616a2f80c6 None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.542367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ad538-5c73-46dd-8ef1-6a616a2f80c6 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.542713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ad538-5c73-46dd-8ef1-6a616a2f80c6 None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/bbfe92babca048d882cb59cb29b3ed97/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.546575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277ad538-5c73-46dd-8ef1-6a616a2f80c6 None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=bbfe92babca048d882cb59cb29b3ed97, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:54.548419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277ad538-5c73-46dd-8ef1-6a616a2f80c6 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:54.565002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 407/804] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:34:54 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/bbfe92babca048d882cb59cb29b3ed97/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:54.565002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277ad538-5c73-46dd-8ef1-6a616a2f80c6 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:54.699694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277ad538-5c73-46dd-8ef1-6a616a2f80c6 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:54.719428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2db9a0-514e-4cbe-a12b-429880ae3db0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:54.746943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57059c5e-7a9c-45f7-bce7-e3e43d7a6f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:54.747652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57059c5e-7a9c-45f7-bce7-e3e43d7a6f80 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.749493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57059c5e-7a9c-45f7-bce7-e3e43d7a6f80 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.749894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57059c5e-7a9c-45f7-bce7-e3e43d7a6f80 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.751557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57059c5e-7a9c-45f7-bce7-e3e43d7a6f80 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:54.753201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57059c5e-7a9c-45f7-bce7-e3e43d7a6f80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:54.754728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57059c5e-7a9c-45f7-bce7-e3e43d7a6f80 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:54.777618 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 408/805] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:54.783968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57059c5e-7a9c-45f7-bce7-e3e43d7a6f80 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:54.795904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63adb10-516a-408f-bc5c-7cc52dcb8108 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:54.796563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63adb10-516a-408f-bc5c-7cc52dcb8108 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.797089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63adb10-516a-408f-bc5c-7cc52dcb8108 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.797975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63adb10-516a-408f-bc5c-7cc52dcb8108 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.810456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63adb10-516a-408f-bc5c-7cc52dcb8108 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:54.811419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-f63adb10-516a-408f-bc5c-7cc52dcb8108 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:54.818496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:54.820241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:54.822374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 398/806] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:34:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:54.833860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63adb10-516a-408f-bc5c-7cc52dcb8108 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:54.848383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:54.905514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57059c5e-7a9c-45f7-bce7-e3e43d7a6f80 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:54.944517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd4cd64-93aa-4f94-a0f3-07a67f145660 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:54.945553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd4cd64-93aa-4f94-a0f3-07a67f145660 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.945837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd4cd64-93aa-4f94-a0f3-07a67f145660 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.946514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd4cd64-93aa-4f94-a0f3-07a67f145660 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.949311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63adb10-516a-408f-bc5c-7cc52dcb8108 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:54.960441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd4cd64-93aa-4f94-a0f3-07a67f145660 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:54.962556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd4cd64-93aa-4f94-a0f3-07a67f145660 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-7dd4cd64-93aa-4f94-a0f3-07a67f145660 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:54.981571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:54.983640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:54.984835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:54.984835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:54.984835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:54.984835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:54.984835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 409/807] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:54.984835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7fc0e3-d44f-4728-a2e0-fc2713b378fc None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.984835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7fc0e3-d44f-4728-a2e0-fc2713b378fc None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.985553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc7fc0e3-d44f-4728-a2e0-fc2713b378fc None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.987476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 410/808] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:54.992261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2062de-3664-4e6c-8cea-a3a168473504 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.992543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2062de-3664-4e6c-8cea-a3a168473504 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.992866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2062de-3664-4e6c-8cea-a3a168473504 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.993313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc307a2-16ad-4b49-8b88-a32549a47342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:54.994067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc307a2-16ad-4b49-8b88-a32549a47342 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:54.994456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc307a2-16ad-4b49-8b88-a32549a47342 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:54.994833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc307a2-16ad-4b49-8b88-a32549a47342 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:54.995685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc307a2-16ad-4b49-8b88-a32549a47342 None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:54.996097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc307a2-16ad-4b49-8b88-a32549a47342 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:54.997653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc307a2-16ad-4b49-8b88-a32549a47342 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:55.008260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 399/809] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:34:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:55.013466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-df2062de-3664-4e6c-8cea-a3a168473504 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:55.013770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc307a2-16ad-4b49-8b88-a32549a47342 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:55.014773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-df2062de-3664-4e6c-8cea-a3a168473504 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:55.092255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:55.114351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 411/810] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:34:55.116052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2062de-3664-4e6c-8cea-a3a168473504 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:55.119393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:55.136224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc307a2-16ad-4b49-8b88-a32549a47342 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:55.174469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087ae435-70fb-497a-837c-46a3baf2f7f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:55.175249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087ae435-70fb-497a-837c-46a3baf2f7f1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:55.175540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087ae435-70fb-497a-837c-46a3baf2f7f1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:55.175820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087ae435-70fb-497a-837c-46a3baf2f7f1 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:55.176530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087ae435-70fb-497a-837c-46a3baf2f7f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:55.176887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087ae435-70fb-497a-837c-46a3baf2f7f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:55.178380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087ae435-70fb-497a-837c-46a3baf2f7f1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:55.197787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 400/811] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:55.198650 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087ae435-70fb-497a-837c-46a3baf2f7f1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:55.250455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2062de-3664-4e6c-8cea-a3a168473504 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:55.286518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c6712d-8d5c-4d54-a40b-a66f22642418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:55.287168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c6712d-8d5c-4d54-a40b-a66f22642418 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:55.287617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c6712d-8d5c-4d54-a40b-a66f22642418 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:55.287901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c6712d-8d5c-4d54-a40b-a66f22642418 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:55.301106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c6712d-8d5c-4d54-a40b-a66f22642418 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:55.303047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c6712d-8d5c-4d54-a40b-a66f22642418 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:55.308448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087ae435-70fb-497a-837c-46a3baf2f7f1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-93c6712d-8d5c-4d54-a40b-a66f22642418 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:55.321382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:55.322291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:55.323271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:55.323271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:55.323271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:55.323271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:55.324880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 412/812] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:55.325773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c6712d-8d5c-4d54-a40b-a66f22642418 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:55.348725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:55.355589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f014638-1d35-47ab-9778-72c936d1dab2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:55.356558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f014638-1d35-47ab-9778-72c936d1dab2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:55.356893 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f014638-1d35-47ab-9778-72c936d1dab2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:55.357175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f014638-1d35-47ab-9778-72c936d1dab2 None admin] PATH_INFO: `/v3/services/network` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:55.358144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f014638-1d35-47ab-9778-72c936d1dab2 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:55.360495 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f014638-1d35-47ab-9778-72c936d1dab2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-0f014638-1d35-47ab-9778-72c936d1dab2 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:55.372057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:55.373192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:55.374039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 401/813] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:55 2022] GET /identity/v3/services/network => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:55.374837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f014638-1d35-47ab-9778-72c936d1dab2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:55.394974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:55.449395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c6712d-8d5c-4d54-a40b-a66f22642418 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:55.488760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6d2572-f430-41b9-8cfe-8dfb9f562572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:55.489839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6d2572-f430-41b9-8cfe-8dfb9f562572 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:55.490320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6d2572-f430-41b9-8cfe-8dfb9f562572 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:55.490855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6d2572-f430-41b9-8cfe-8dfb9f562572 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:55.499458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6d2572-f430-41b9-8cfe-8dfb9f562572 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:55.502054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6d2572-f430-41b9-8cfe-8dfb9f562572 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:55.514085 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f014638-1d35-47ab-9778-72c936d1dab2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-1a6d2572-f430-41b9-8cfe-8dfb9f562572 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:55.515280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:55.516968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:55.517901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:55.517901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:55.517901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:55.517901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:55.517901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:55.517901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:55.517901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:55.517901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:55.517901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:34:55.517901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:55.520617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 413/814] 10.222.0.40 () {56 vars in 1093 bytes} [Fri Jul 29 04:34:55 2022] GET /identity/v3/users/nova => generated 82 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:55.521609 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6d2572-f430-41b9-8cfe-8dfb9f562572 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:55.567927 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d85211a-3e6a-44d3-86b7-f79b4d4a6b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:55.570640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d85211a-3e6a-44d3-86b7-f79b4d4a6b32 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:55.570640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d85211a-3e6a-44d3-86b7-f79b4d4a6b32 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:55.570640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d85211a-3e6a-44d3-86b7-f79b4d4a6b32 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:55.570640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d85211a-3e6a-44d3-86b7-f79b4d4a6b32 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:55.576856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d85211a-3e6a-44d3-86b7-f79b4d4a6b32 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:55.576856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d85211a-3e6a-44d3-86b7-f79b4d4a6b32 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:55.592821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 402/815] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:55.593698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d85211a-3e6a-44d3-86b7-f79b4d4a6b32 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:55.658421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6d2572-f430-41b9-8cfe-8dfb9f562572 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:55.670141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:55.678546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:55.709088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11253202-c6cd-4d17-b7c7-308030ecedb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:55.709722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11253202-c6cd-4d17-b7c7-308030ecedb5 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:55.710008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11253202-c6cd-4d17-b7c7-308030ecedb5 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:55.710290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11253202-c6cd-4d17-b7c7-308030ecedb5 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:55.711256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11253202-c6cd-4d17-b7c7-308030ecedb5 None admin] RBAC: Adding query filter params (name=network) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:55.711567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11253202-c6cd-4d17-b7c7-308030ecedb5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:55.713508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11253202-c6cd-4d17-b7c7-308030ecedb5 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:55.724747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 414/816] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:34:55 2022] GET /identity/v3/services?name=network => generated 124 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:55.724747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11253202-c6cd-4d17-b7c7-308030ecedb5 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:55.727165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d85211a-3e6a-44d3-86b7-f79b4d4a6b32 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:55.774932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cacae0b-1029-4590-b61b-22dd943c5d53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:55.775475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cacae0b-1029-4590-b61b-22dd943c5d53 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:55.775762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cacae0b-1029-4590-b61b-22dd943c5d53 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:55.776041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cacae0b-1029-4590-b61b-22dd943c5d53 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:55.777295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cacae0b-1029-4590-b61b-22dd943c5d53 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:55.777612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cacae0b-1029-4590-b61b-22dd943c5d53 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:55.779355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cacae0b-1029-4590-b61b-22dd943c5d53 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:55.808758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 403/817] 10.222.0.40 () {56 vars in 1133 bytes} [Fri Jul 29 04:34:55 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:55.809958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cacae0b-1029-4590-b61b-22dd943c5d53 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:55.848992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11253202-c6cd-4d17-b7c7-308030ecedb5 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:55.888010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41c98de-a8e0-43c4-a9d2-9b2e85b7e5b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:55.889278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41c98de-a8e0-43c4-a9d2-9b2e85b7e5b1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:55.889670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41c98de-a8e0-43c4-a9d2-9b2e85b7e5b1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:55.890057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41c98de-a8e0-43c4-a9d2-9b2e85b7e5b1 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:55.900068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41c98de-a8e0-43c4-a9d2-9b2e85b7e5b1 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:55.901719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41c98de-a8e0-43c4-a9d2-9b2e85b7e5b1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-d41c98de-a8e0-43c4-a9d2-9b2e85b7e5b1 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:55.915115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:55.916830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:55.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:55.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:55.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:55.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:55.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:55.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:55.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:55.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:55.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:34:55.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:55.923830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 415/818] 10.222.0.40 () {56 vars in 1117 bytes} [Fri Jul 29 04:34:55 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:55.923830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41c98de-a8e0-43c4-a9d2-9b2e85b7e5b1 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:55.956326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cacae0b-1029-4590-b61b-22dd943c5d53 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:55.971241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:55.982908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:55.997333 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc09707f-9258-481e-8398-5a2ac17fef73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:55.998147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc09707f-9258-481e-8398-5a2ac17fef73 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:55.998511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc09707f-9258-481e-8398-5a2ac17fef73 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:55.998871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc09707f-9258-481e-8398-5a2ac17fef73 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:55.999735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc09707f-9258-481e-8398-5a2ac17fef73 None admin] RBAC: Adding query filter params (type=network) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:56.000152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc09707f-9258-481e-8398-5a2ac17fef73 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:56.001767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc09707f-9258-481e-8398-5a2ac17fef73 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:56.011670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 404/819] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:34:55 2022] GET /identity/v3/services?type=network => generated 350 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:56.012752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc09707f-9258-481e-8398-5a2ac17fef73 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:56.055323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41c98de-a8e0-43c4-a9d2-9b2e85b7e5b1 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:56.091918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329ec28e-7cc2-4594-bd86-5229f863e91a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:56.092687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329ec28e-7cc2-4594-bd86-5229f863e91a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.093048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329ec28e-7cc2-4594-bd86-5229f863e91a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.093396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329ec28e-7cc2-4594-bd86-5229f863e91a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.132011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc09707f-9258-481e-8398-5a2ac17fef73 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:56.166314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329ec28e-7cc2-4594-bd86-5229f863e91a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:56.167929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329ec28e-7cc2-4594-bd86-5229f863e91a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:56.171389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682bc2e9-00e4-4400-a9c2-359cfd8cf198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:56.172350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682bc2e9-00e4-4400-a9c2-359cfd8cf198 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.173274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682bc2e9-00e4-4400-a9c2-359cfd8cf198 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.173835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682bc2e9-00e4-4400-a9c2-359cfd8cf198 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.174814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682bc2e9-00e4-4400-a9c2-359cfd8cf198 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:56.175354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682bc2e9-00e4-4400-a9c2-359cfd8cf198 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:56.177017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682bc2e9-00e4-4400-a9c2-359cfd8cf198 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:56.201377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 405/820] 10.222.0.40 () {56 vars in 1149 bytes} [Fri Jul 29 04:34:56 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:56.202619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682bc2e9-00e4-4400-a9c2-359cfd8cf198 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:56.211442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:56.241957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:56.284126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 416/821] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:55 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:34:56.284126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329ec28e-7cc2-4594-bd86-5229f863e91a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:56.321986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682bc2e9-00e4-4400-a9c2-359cfd8cf198 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:56.359694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a2e726-5c7a-4fbd-921b-748d2f39ee31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:56.360942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a2e726-5c7a-4fbd-921b-748d2f39ee31 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.361435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a2e726-5c7a-4fbd-921b-748d2f39ee31 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.361931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a2e726-5c7a-4fbd-921b-748d2f39ee31 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.362841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a2e726-5c7a-4fbd-921b-748d2f39ee31 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:56.364434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a2e726-5c7a-4fbd-921b-748d2f39ee31 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:56.388349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329ec28e-7cc2-4594-bd86-5229f863e91a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:56.393793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 406/822] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:34:56 2022] POST /identity/v3/endpoints => generated 327 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:34:56.394938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a2e726-5c7a-4fbd-921b-748d2f39ee31 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:56.516652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:56.517515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194a1c37-af17-45a7-b806-8dac777d3c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:56.518120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a1c37-af17-45a7-b806-8dac777d3c6a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.518428 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a1c37-af17-45a7-b806-8dac777d3c6a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.518688 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194a1c37-af17-45a7-b806-8dac777d3c6a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.521273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:56.532226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a2e726-5c7a-4fbd-921b-748d2f39ee31 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:56.543654 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194a1c37-af17-45a7-b806-8dac777d3c6a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:56.545193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194a1c37-af17-45a7-b806-8dac777d3c6a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-194a1c37-af17-45a7-b806-8dac777d3c6a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:56.569064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:56.570047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:56.570909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:56.570909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:56.570909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:56.570909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:56.577843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 417/823] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:56.579429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680b1541-cc82-460a-b78f-472b855be709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:56.580094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680b1541-cc82-460a-b78f-472b855be709 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.580377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680b1541-cc82-460a-b78f-472b855be709 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.580700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680b1541-cc82-460a-b78f-472b855be709 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.581304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1d7674-e964-4ab2-bc5a-c1c7798c848e None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.581833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1d7674-e964-4ab2-bc5a-c1c7798c848e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.582312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1d7674-e964-4ab2-bc5a-c1c7798c848e None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.588482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 418/824] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:56 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:56.589972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1d7674-e964-4ab2-bc5a-c1c7798c848e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:56.594334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680b1541-cc82-460a-b78f-472b855be709 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:56.597078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680b1541-cc82-460a-b78f-472b855be709 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-680b1541-cc82-460a-b78f-472b855be709 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:56.617644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:56.618575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:56.619478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:56.619478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:56.619478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:56.619478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:56.619478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 407/825] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:56.624843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf9965a-9f8f-465e-974f-70ee6e551727 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.624843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf9965a-9f8f-465e-974f-70ee6e551727 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.624843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf9965a-9f8f-465e-974f-70ee6e551727 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.646325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-baf9965a-9f8f-465e-974f-70ee6e551727 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:56.647532 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-baf9965a-9f8f-465e-974f-70ee6e551727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:56.726839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1d7674-e964-4ab2-bc5a-c1c7798c848e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:56.741758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 408/826] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:56 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:34:56.742607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf9965a-9f8f-465e-974f-70ee6e551727 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:56.766865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd447a9-3cf5-4a93-96a7-9b26ea496102 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:56.767770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd447a9-3cf5-4a93-96a7-9b26ea496102 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.768209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd447a9-3cf5-4a93-96a7-9b26ea496102 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.768663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd447a9-3cf5-4a93-96a7-9b26ea496102 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.769510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd447a9-3cf5-4a93-96a7-9b26ea496102 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:56.769969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd447a9-3cf5-4a93-96a7-9b26ea496102 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:56.771506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd447a9-3cf5-4a93-96a7-9b26ea496102 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:56.788161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 419/827] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:56.789264 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd447a9-3cf5-4a93-96a7-9b26ea496102 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:56.826625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:56.838480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:56.858092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf9965a-9f8f-465e-974f-70ee6e551727 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:56.891487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd447a9-3cf5-4a93-96a7-9b26ea496102 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:56.892868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150043f1-30d5-40da-86e7-be1f9cf49e30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:56.893485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150043f1-30d5-40da-86e7-be1f9cf49e30 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.893776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150043f1-30d5-40da-86e7-be1f9cf49e30 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.894055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150043f1-30d5-40da-86e7-be1f9cf49e30 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.894752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150043f1-30d5-40da-86e7-be1f9cf49e30 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:56.895061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150043f1-30d5-40da-86e7-be1f9cf49e30 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:56.896444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150043f1-30d5-40da-86e7-be1f9cf49e30 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:56.907426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 409/828] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:56.908247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150043f1-30d5-40da-86e7-be1f9cf49e30 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:56.934121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a9367a-b8bb-496a-899d-40f7a9dc8eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:56.934758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a9367a-b8bb-496a-899d-40f7a9dc8eec None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.935044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a9367a-b8bb-496a-899d-40f7a9dc8eec None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.935331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a9367a-b8bb-496a-899d-40f7a9dc8eec None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.935957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a9367a-b8bb-496a-899d-40f7a9dc8eec None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:56.937362 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a9367a-b8bb-496a-899d-40f7a9dc8eec None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-a9a9367a-b8bb-496a-899d-40f7a9dc8eec None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:56.946217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:56.947359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:56.948319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 420/829] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:34:56 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:56.949858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e1cde1-8086-47aa-a431-5f30550df392 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:56.950153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e1cde1-8086-47aa-a431-5f30550df392 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:56.950439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e1cde1-8086-47aa-a431-5f30550df392 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:56.951824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 421/830] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:56.952394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e1cde1-8086-47aa-a431-5f30550df392 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:57.012159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150043f1-30d5-40da-86e7-be1f9cf49e30 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:57.041256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:57.046739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fe8a57-db29-4a26-8a19-96f125950f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:57.048886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fe8a57-db29-4a26-8a19-96f125950f89 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.049168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fe8a57-db29-4a26-8a19-96f125950f89 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.049451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fe8a57-db29-4a26-8a19-96f125950f89 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.054950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e1cde1-8086-47aa-a431-5f30550df392 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:57.056056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fe8a57-db29-4a26-8a19-96f125950f89 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:57.057543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fe8a57-db29-4a26-8a19-96f125950f89 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:57.065572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-e8fe8a57-db29-4a26-8a19-96f125950f89 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:57.069497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:57.070554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:57.076545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:57.076545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:57.076545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:57.076545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:57.076545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:57.076545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:57.076545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:57.076545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:57.076545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 410/831] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:57.076545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fe8a57-db29-4a26-8a19-96f125950f89 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:57.090201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c16c08-9c50-414e-a208-2776fd477f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:57.090777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c16c08-9c50-414e-a208-2776fd477f32 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.091060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c16c08-9c50-414e-a208-2776fd477f32 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.091336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c16c08-9c50-414e-a208-2776fd477f32 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.099103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c16c08-9c50-414e-a208-2776fd477f32 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:57.100546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c16c08-9c50-414e-a208-2776fd477f32 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-68c16c08-9c50-414e-a208-2776fd477f32 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:57.112803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:57.113830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 422/832] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49500fd8-723b-409a-b12f-431c69735ee8 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.119736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49500fd8-723b-409a-b12f-431c69735ee8 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.120474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49500fd8-723b-409a-b12f-431c69735ee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.133238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-49500fd8-723b-409a-b12f-431c69735ee8 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:57.134532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-49500fd8-723b-409a-b12f-431c69735ee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:57.180285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fe8a57-db29-4a26-8a19-96f125950f89 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:57.201947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 423/833] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:57 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:34:57.202773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49500fd8-723b-409a-b12f-431c69735ee8 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:57.223165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7fdad-5b8a-435d-b1e5-2ec4f24dbe6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:57.223985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7fdad-5b8a-435d-b1e5-2ec4f24dbe6e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.224339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7fdad-5b8a-435d-b1e5-2ec4f24dbe6e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.224645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d7fdad-5b8a-435d-b1e5-2ec4f24dbe6e None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.225348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d7fdad-5b8a-435d-b1e5-2ec4f24dbe6e None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:57.225658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d7fdad-5b8a-435d-b1e5-2ec4f24dbe6e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:57.227113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d7fdad-5b8a-435d-b1e5-2ec4f24dbe6e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:57.231762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 411/834] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:34:57 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:57.232538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7fdad-5b8a-435d-b1e5-2ec4f24dbe6e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:57.287537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:57.299519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:57.302790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49500fd8-723b-409a-b12f-431c69735ee8 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:57.331693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d7fdad-5b8a-435d-b1e5-2ec4f24dbe6e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:57.336476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9e81ba-9eeb-442a-9350-2e81aed808af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:57.337118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9e81ba-9eeb-442a-9350-2e81aed808af None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.337420 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9e81ba-9eeb-442a-9350-2e81aed808af None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.337684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9e81ba-9eeb-442a-9350-2e81aed808af None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.338428 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9e81ba-9eeb-442a-9350-2e81aed808af None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:57.338753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9e81ba-9eeb-442a-9350-2e81aed808af None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:57.340155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9e81ba-9eeb-442a-9350-2e81aed808af None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:57.351685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 424/835] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:57.352405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9e81ba-9eeb-442a-9350-2e81aed808af None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:57.370904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d531bca2-9fb1-49b3-9e43-3861078588c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:57.371534 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d531bca2-9fb1-49b3-9e43-3861078588c9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.371825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d531bca2-9fb1-49b3-9e43-3861078588c9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.372125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d531bca2-9fb1-49b3-9e43-3861078588c9 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.372898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d531bca2-9fb1-49b3-9e43-3861078588c9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:57.373214 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d531bca2-9fb1-49b3-9e43-3861078588c9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:57.374648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d531bca2-9fb1-49b3-9e43-3861078588c9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:57.387557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 412/836] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:57.388362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d531bca2-9fb1-49b3-9e43-3861078588c9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:57.455288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9e81ba-9eeb-442a-9350-2e81aed808af None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:57.487219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d531bca2-9fb1-49b3-9e43-3861078588c9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:57.489341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe550fc-54c5-4f92-a19b-8af61ddbb78f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:57.489987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe550fc-54c5-4f92-a19b-8af61ddbb78f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.490276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe550fc-54c5-4f92-a19b-8af61ddbb78f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.490553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe550fc-54c5-4f92-a19b-8af61ddbb78f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.501185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe550fc-54c5-4f92-a19b-8af61ddbb78f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:34:57.504110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-6fe550fc-54c5-4f92-a19b-8af61ddbb78f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:34:57.506514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:57.507623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:57.508716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 425/837] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:57.509500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe550fc-54c5-4f92-a19b-8af61ddbb78f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:57.527599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14bf8dc-abc8-4af6-bfaf-8d679c30c3f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:57.528228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14bf8dc-abc8-4af6-bfaf-8d679c30c3f3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.528518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14bf8dc-abc8-4af6-bfaf-8d679c30c3f3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.528844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14bf8dc-abc8-4af6-bfaf-8d679c30c3f3 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.529534 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14bf8dc-abc8-4af6-bfaf-8d679c30c3f3 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:57.529849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14bf8dc-abc8-4af6-bfaf-8d679c30c3f3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:57.530223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:57.531250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14bf8dc-abc8-4af6-bfaf-8d679c30c3f3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:57.535757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 413/838] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:34:57 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:57.536543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14bf8dc-abc8-4af6-bfaf-8d679c30c3f3 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:57.630235 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe550fc-54c5-4f92-a19b-8af61ddbb78f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:57.653884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14bf8dc-abc8-4af6-bfaf-8d679c30c3f3 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:57.674268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3411fe1b-0ec5-4e9e-a32e-632ca0ffa752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:57.674969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3411fe1b-0ec5-4e9e-a32e-632ca0ffa752 None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.675265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3411fe1b-0ec5-4e9e-a32e-632ca0ffa752 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.675527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3411fe1b-0ec5-4e9e-a32e-632ca0ffa752 None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/3f53019d35f244e4959d8340ba6f7e45/roles/7140c1d0b4f74727a1885faa13c9e6ec` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.693677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3411fe1b-0ec5-4e9e-a32e-632ca0ffa752 None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=3f53019d35f244e4959d8340ba6f7e45, role_id=7140c1d0b4f74727a1885faa13c9e6ec)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:57.695252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3411fe1b-0ec5-4e9e-a32e-632ca0ffa752 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:57.702238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800121db-97fe-4a51-b84f-5300ce8ad9ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:57.702876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800121db-97fe-4a51-b84f-5300ce8ad9ec None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.703195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800121db-97fe-4a51-b84f-5300ce8ad9ec None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.703710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800121db-97fe-4a51-b84f-5300ce8ad9ec None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.704354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800121db-97fe-4a51-b84f-5300ce8ad9ec None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=441bcfe715d44efea69d556677094857, user.id=58c33a72a871432b88da4f0bf1bc514c) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:57.704698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800121db-97fe-4a51-b84f-5300ce8ad9ec None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:57.706826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800121db-97fe-4a51-b84f-5300ce8ad9ec None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:57.713384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 414/839] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:57 2022] GET /identity/v3/role_assignments?user.id=58c33a72a871432b88da4f0bf1bc514c&scope.project.id=441bcfe715d44efea69d556677094857&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:57.713539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 426/840] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:34:57 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/3f53019d35f244e4959d8340ba6f7e45/roles/7140c1d0b4f74727a1885faa13c9e6ec => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:34:57.713634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800121db-97fe-4a51-b84f-5300ce8ad9ec None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:57.775285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:57.848627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800121db-97fe-4a51-b84f-5300ce8ad9ec None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:57.889904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb352e35-1103-4b35-9e64-946b2970b1db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:57.890462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb352e35-1103-4b35-9e64-946b2970b1db None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:57.890795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb352e35-1103-4b35-9e64-946b2970b1db None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:57.891238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb352e35-1103-4b35-9e64-946b2970b1db None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:57.892040 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb352e35-1103-4b35-9e64-946b2970b1db None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:57.892357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb352e35-1103-4b35-9e64-946b2970b1db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:57.893886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb352e35-1103-4b35-9e64-946b2970b1db None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:57.903010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 415/841] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:34:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:57.909527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3411fe1b-0ec5-4e9e-a32e-632ca0ffa752 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:57.982873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:58.016253 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3411fe1b-0ec5-4e9e-a32e-632ca0ffa752 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:58.060489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0584beb-a24b-47d3-b41f-92d33eb5f134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:58.061255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0584beb-a24b-47d3-b41f-92d33eb5f134 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:58.061551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0584beb-a24b-47d3-b41f-92d33eb5f134 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:58.061835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0584beb-a24b-47d3-b41f-92d33eb5f134 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:58.073060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0584beb-a24b-47d3-b41f-92d33eb5f134 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:58.074465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0584beb-a24b-47d3-b41f-92d33eb5f134 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-b0584beb-a24b-47d3-b41f-92d33eb5f134 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:58.090810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:58.091976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:58.092891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:58.092891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:58.092891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:58.092891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:58.092891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 427/842] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:58.101759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb352e35-1103-4b35-9e64-946b2970b1db None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:58.208912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb352e35-1103-4b35-9e64-946b2970b1db None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:58.217232 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:58.247031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6b7519-f6da-46bb-811d-9e5f9ec680bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:58.247849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6b7519-f6da-46bb-811d-9e5f9ec680bf None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:58.248135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6b7519-f6da-46bb-811d-9e5f9ec680bf None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:58.248458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6b7519-f6da-46bb-811d-9e5f9ec680bf None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:58.249198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6b7519-f6da-46bb-811d-9e5f9ec680bf None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:58.249504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6b7519-f6da-46bb-811d-9e5f9ec680bf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:58.250908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6b7519-f6da-46bb-811d-9e5f9ec680bf None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:58.264079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 416/843] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:58.270024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0584beb-a24b-47d3-b41f-92d33eb5f134 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:58.381016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0584beb-a24b-47d3-b41f-92d33eb5f134 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:58.405091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:58.418309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77815af0-5a94-4c08-b087-5856c3e2e895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:58.418959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77815af0-5a94-4c08-b087-5856c3e2e895 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:58.419247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77815af0-5a94-4c08-b087-5856c3e2e895 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:58.419531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77815af0-5a94-4c08-b087-5856c3e2e895 None admin] PATH_INFO: `/v3/users/swift` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:58.427777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77815af0-5a94-4c08-b087-5856c3e2e895 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:58.429276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77815af0-5a94-4c08-b087-5856c3e2e895 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-77815af0-5a94-4c08-b087-5856c3e2e895 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:34:58.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:58.442717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:34:58.443650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 428/844] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:34:58 2022] GET /identity/v3/users/swift => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:58.448715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6b7519-f6da-46bb-811d-9e5f9ec680bf None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:58.548385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6b7519-f6da-46bb-811d-9e5f9ec680bf None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:58.589228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef21a837-d271-4127-b67e-d9be06c096f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:58.589858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef21a837-d271-4127-b67e-d9be06c096f7 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:58.590199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef21a837-d271-4127-b67e-d9be06c096f7 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:58.590479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef21a837-d271-4127-b67e-d9be06c096f7 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:58.591275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef21a837-d271-4127-b67e-d9be06c096f7 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:58.591584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef21a837-d271-4127-b67e-d9be06c096f7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:58.593044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef21a837-d271-4127-b67e-d9be06c096f7 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:58.610712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 417/845] 10.222.0.40 () {56 vars in 1135 bytes} [Fri Jul 29 04:34:58 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:58.616335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77815af0-5a94-4c08-b087-5856c3e2e895 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:58.739869 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77815af0-5a94-4c08-b087-5856c3e2e895 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:58.776312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9144c709-6148-4262-a464-8992f5c352cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:58.776998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9144c709-6148-4262-a464-8992f5c352cc None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:58.777285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9144c709-6148-4262-a464-8992f5c352cc None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:58.777561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9144c709-6148-4262-a464-8992f5c352cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:58.831703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:58.855525 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9144c709-6148-4262-a464-8992f5c352cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:58.857031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9144c709-6148-4262-a464-8992f5c352cc None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:58.964346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 429/846] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:34:58 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:34:58.969866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef21a837-d271-4127-b67e-d9be06c096f7 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:59.004965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:59.082516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef21a837-d271-4127-b67e-d9be06c096f7 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:59.119824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af88da-dded-45ec-acf2-77e8af020f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:59.120772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af88da-dded-45ec-acf2-77e8af020f8c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:59.121204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af88da-dded-45ec-acf2-77e8af020f8c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:59.121628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24af88da-dded-45ec-acf2-77e8af020f8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:59.133309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24af88da-dded-45ec-acf2-77e8af020f8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:59.134935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24af88da-dded-45ec-acf2-77e8af020f8c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-24af88da-dded-45ec-acf2-77e8af020f8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:34:59.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:59.151678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:59.152786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:34:59.152786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:34:59.152786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:34:59.152786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:59.154583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 418/847] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:34:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:59.161943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9144c709-6148-4262-a464-8992f5c352cc None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:59.245378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:59.269041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9144c709-6148-4262-a464-8992f5c352cc None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:59.305330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf276c6-5d68-40b0-b5c0-7f3b8459a88a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:59.306125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf276c6-5d68-40b0-b5c0-7f3b8459a88a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:59.306512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf276c6-5d68-40b0-b5c0-7f3b8459a88a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:59.306894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf276c6-5d68-40b0-b5c0-7f3b8459a88a None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:59.307718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf276c6-5d68-40b0-b5c0-7f3b8459a88a None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:59.308129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf276c6-5d68-40b0-b5c0-7f3b8459a88a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:59.309711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf276c6-5d68-40b0-b5c0-7f3b8459a88a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:59.322590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 430/848] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:34:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:59.327958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af88da-dded-45ec-acf2-77e8af020f8c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:59.411370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:59.434983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24af88da-dded-45ec-acf2-77e8af020f8c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:59.472713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45391183-e2db-417a-934a-e1a566dc5153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:59.473371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45391183-e2db-417a-934a-e1a566dc5153 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:59.473899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45391183-e2db-417a-934a-e1a566dc5153 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:59.474253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45391183-e2db-417a-934a-e1a566dc5153 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:59.481466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45391183-e2db-417a-934a-e1a566dc5153 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:59.483006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45391183-e2db-417a-934a-e1a566dc5153 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-45391183-e2db-417a-934a-e1a566dc5153 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:34:59.496017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:34:59.497128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:34:59.498212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:34:59.498212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:34:59.498212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:34:59.498212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:34:59.498212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:34:59.498212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:34:59.498212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:34:59.498212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:34:59.499487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 419/849] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:34:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:34:59.505924 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf276c6-5d68-40b0-b5c0-7f3b8459a88a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:59.615782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf276c6-5d68-40b0-b5c0-7f3b8459a88a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:59.628948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5549ff00-3d70-46a7-a977-c19e22dcdeb2 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:59.629443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5549ff00-3d70-46a7-a977-c19e22dcdeb2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:59.629557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5549ff00-3d70-46a7-a977-c19e22dcdeb2 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:59.631018 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 420/850] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:59.635894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097ba839-9f91-4778-9b51-675fdfa862f1 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:59.636185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097ba839-9f91-4778-9b51-675fdfa862f1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:59.636465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097ba839-9f91-4778-9b51-675fdfa862f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:59.658945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38881ae0-5eea-4054-86a3-8cc6c56ecdff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:59.659589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38881ae0-5eea-4054-86a3-8cc6c56ecdff None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:59.659879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38881ae0-5eea-4054-86a3-8cc6c56ecdff None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:59.660163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38881ae0-5eea-4054-86a3-8cc6c56ecdff None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:59.660968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38881ae0-5eea-4054-86a3-8cc6c56ecdff None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:59.661284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38881ae0-5eea-4054-86a3-8cc6c56ecdff None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:59.662278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-097ba839-9f91-4778-9b51-675fdfa862f1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:59.662728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38881ae0-5eea-4054-86a3-8cc6c56ecdff None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:59.663955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-097ba839-9f91-4778-9b51-675fdfa862f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:59.680746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 431/851] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:34:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:34:59.683838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38881ae0-5eea-4054-86a3-8cc6c56ecdff None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:59.686944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:59.742487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 421/852] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:59 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:34:59.766983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097ba839-9f91-4778-9b51-675fdfa862f1 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:34:59.803454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38881ae0-5eea-4054-86a3-8cc6c56ecdff None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:59.839286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:59.847320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a67ec7-dc47-40a7-9e10-dff5978b2e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:59.847941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a67ec7-dc47-40a7-9e10-dff5978b2e8f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:59.848222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a67ec7-dc47-40a7-9e10-dff5978b2e8f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:59.848498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a67ec7-dc47-40a7-9e10-dff5978b2e8f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:59.849408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a67ec7-dc47-40a7-9e10-dff5978b2e8f None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=441bcfe715d44efea69d556677094857, user.id=bbfe92babca048d882cb59cb29b3ed97) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:34:59.849624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a67ec7-dc47-40a7-9e10-dff5978b2e8f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:59.851066 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a67ec7-dc47-40a7-9e10-dff5978b2e8f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:59.864973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 432/853] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:34:59 2022] GET /identity/v3/role_assignments?user.id=bbfe92babca048d882cb59cb29b3ed97&scope.project.id=441bcfe715d44efea69d556677094857&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:34:59.866365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481467e9-ba5a-435d-9fba-ce4c9f52e128 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:59.866672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481467e9-ba5a-435d-9fba-ce4c9f52e128 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:59.866923 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481467e9-ba5a-435d-9fba-ce4c9f52e128 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:59.868643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 433/854] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:34:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:34:59.874134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32a2f50-b9de-4257-a021-196615cbc7e2 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:59.874421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32a2f50-b9de-4257-a021-196615cbc7e2 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:59.874698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32a2f50-b9de-4257-a021-196615cbc7e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:59.878404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097ba839-9f91-4778-9b51-675fdfa862f1 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:34:59.897520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c32a2f50-b9de-4257-a021-196615cbc7e2 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:34:59.899231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c32a2f50-b9de-4257-a021-196615cbc7e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:34:59.932832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ee262b-d721-46ff-9f0c-5efb7c70d502 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:34:59.933937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ee262b-d721-46ff-9f0c-5efb7c70d502 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:34:59.934439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ee262b-d721-46ff-9f0c-5efb7c70d502 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:34:59.934912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ee262b-d721-46ff-9f0c-5efb7c70d502 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:34:59.935837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ee262b-d721-46ff-9f0c-5efb7c70d502 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:34:59.937675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ee262b-d721-46ff-9f0c-5efb7c70d502 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:34:59.958101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 422/855] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:34:59 2022] POST /identity/v3/services => generated 251 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:34:59.982392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:34:59.999580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 434/856] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:34:59 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:00.025373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ee262b-d721-46ff-9f0c-5efb7c70d502 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:00.074386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4a4a0-850d-42c1-9a8c-d857b66d87fe None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:00.074386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4a4a0-850d-42c1-9a8c-d857b66d87fe None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:00.074673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4a4a0-850d-42c1-9a8c-d857b66d87fe None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:00.077860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 435/857] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:35:00.083635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b4c6c7-d17f-4afa-98d7-89a681ca2088 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:00.083635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b4c6c7-d17f-4afa-98d7-89a681ca2088 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:00.083878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b4c6c7-d17f-4afa-98d7-89a681ca2088 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:00.105571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-88b4c6c7-d17f-4afa-98d7-89a681ca2088 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:00.106818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-88b4c6c7-d17f-4afa-98d7-89a681ca2088 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:00.149044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ee262b-d721-46ff-9f0c-5efb7c70d502 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:00.178623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:00.189466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 436/858] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:00 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:00.193003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf749eb6-b35a-4c42-b06e-378c3cd2d297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:00.193928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf749eb6-b35a-4c42-b06e-378c3cd2d297 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:00.194411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf749eb6-b35a-4c42-b06e-378c3cd2d297 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:00.194928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf749eb6-b35a-4c42-b06e-378c3cd2d297 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:00.206949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf749eb6-b35a-4c42-b06e-378c3cd2d297 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:00.211114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b4c6c7-d17f-4afa-98d7-89a681ca2088 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-bf749eb6-b35a-4c42-b06e-378c3cd2d297 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:00.212797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:00.213705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:35:00.214540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:00.216257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 423/859] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:35:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:00.222599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf749eb6-b35a-4c42-b06e-378c3cd2d297 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:00.320272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b4c6c7-d17f-4afa-98d7-89a681ca2088 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:00.324759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf749eb6-b35a-4c42-b06e-378c3cd2d297 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:00.345034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:00.358946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d834d0d-9358-4aa1-a9e2-694ff864344b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:00.360070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bcd083-b1ba-42c4-96a0-179a7d88fbab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:00.360247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d834d0d-9358-4aa1-a9e2-694ff864344b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:00.360539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d834d0d-9358-4aa1-a9e2-694ff864344b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:00.360863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d834d0d-9358-4aa1-a9e2-694ff864344b None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:00.361448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bcd083-b1ba-42c4-96a0-179a7d88fbab None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:00.361588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d834d0d-9358-4aa1-a9e2-694ff864344b None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:00.361903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d834d0d-9358-4aa1-a9e2-694ff864344b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:00.362168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bcd083-b1ba-42c4-96a0-179a7d88fbab None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:00.362591 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bcd083-b1ba-42c4-96a0-179a7d88fbab None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:00.363344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d834d0d-9358-4aa1-a9e2-694ff864344b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:00.363515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bcd083-b1ba-42c4-96a0-179a7d88fbab None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:00.365183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bcd083-b1ba-42c4-96a0-179a7d88fbab None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c2bcd083-b1ba-42c4-96a0-179a7d88fbab None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:00.375354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:00.376649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 29 04:35:00.377615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:00.378404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 424/860] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:35:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:00.378404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d834d0d-9358-4aa1-a9e2-694ff864344b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:00.380206 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 437/861] 10.222.0.40 () {56 vars in 1132 bytes} [Fri Jul 29 04:35:00 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:00.388433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bcd083-b1ba-42c4-96a0-179a7d88fbab None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:00.431069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:00.495633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d834d0d-9358-4aa1-a9e2-694ff864344b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:00.509733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bcd083-b1ba-42c4-96a0-179a7d88fbab None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:00.544196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5875729-a771-4fcd-b612-72533bd15637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:00.545017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5875729-a771-4fcd-b612-72533bd15637 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:00.545397 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5875729-a771-4fcd-b612-72533bd15637 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:00.545773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5875729-a771-4fcd-b612-72533bd15637 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:00.546423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb6498d-1444-4ed3-8157-c08ed78e0eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:00.547799 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb6498d-1444-4ed3-8157-c08ed78e0eb9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:00.548437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb6498d-1444-4ed3-8157-c08ed78e0eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:00.548756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb6498d-1444-4ed3-8157-c08ed78e0eb9 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:00.549449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb6498d-1444-4ed3-8157-c08ed78e0eb9 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:00.549760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb6498d-1444-4ed3-8157-c08ed78e0eb9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:00.551145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb6498d-1444-4ed3-8157-c08ed78e0eb9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:00.556336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 438/862] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:00 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:00.560754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5875729-a771-4fcd-b612-72533bd15637 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:00.561673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb6498d-1444-4ed3-8157-c08ed78e0eb9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:00.562549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5875729-a771-4fcd-b612-72533bd15637 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-b5875729-a771-4fcd-b612-72533bd15637 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:00.582400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:00.583601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:00.584772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:00.584772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:00.584772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:00.584772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:00.588690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 425/863] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:00.590172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5875729-a771-4fcd-b612-72533bd15637 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:00.627861 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:00.701373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb6498d-1444-4ed3-8157-c08ed78e0eb9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:00.737829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:00.740951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd44683-2d8d-490f-bdd5-0d705f6ca7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:00.741742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd44683-2d8d-490f-bdd5-0d705f6ca7a8 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:00.742032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd44683-2d8d-490f-bdd5-0d705f6ca7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:00.742313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd44683-2d8d-490f-bdd5-0d705f6ca7a8 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:00.742629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5875729-a771-4fcd-b612-72533bd15637 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:00.743317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd44683-2d8d-490f-bdd5-0d705f6ca7a8 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:00.743629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd44683-2d8d-490f-bdd5-0d705f6ca7a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:00.746649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd44683-2d8d-490f-bdd5-0d705f6ca7a8 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:00.752995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 439/864] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:00 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:00.787418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da54a15-8966-4d39-9073-c2e3ffc369fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:00.788422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da54a15-8966-4d39-9073-c2e3ffc369fc None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:00.788892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da54a15-8966-4d39-9073-c2e3ffc369fc None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:00.789350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da54a15-8966-4d39-9073-c2e3ffc369fc None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:00.790237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da54a15-8966-4d39-9073-c2e3ffc369fc None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:00.790703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da54a15-8966-4d39-9073-c2e3ffc369fc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:00.792343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da54a15-8966-4d39-9073-c2e3ffc369fc None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:00.807285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 426/865] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:00.814315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd44683-2d8d-490f-bdd5-0d705f6ca7a8 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:00.926091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd44683-2d8d-490f-bdd5-0d705f6ca7a8 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:00.941884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:00.963478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2a033c-0e22-43fc-94a4-5b540d142fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:00.964205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2a033c-0e22-43fc-94a4-5b540d142fb5 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:00.964490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2a033c-0e22-43fc-94a4-5b540d142fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:00.964800 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2a033c-0e22-43fc-94a4-5b540d142fb5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:00.973399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2a033c-0e22-43fc-94a4-5b540d142fb5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:00.974909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2a033c-0e22-43fc-94a4-5b540d142fb5 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-2d2a033c-0e22-43fc-94a4-5b540d142fb5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:35:00.986469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:00.987402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:00.988377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:00.988377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:00.988377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:35:00.988377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:35:00.988377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:35:00.988377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:35:00.988377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:35:00.988377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:35:00.988377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:35:00.988377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:00.992265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 440/866] 10.222.0.40 () {56 vars in 1117 bytes} [Fri Jul 29 04:35:00 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:00.993625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da54a15-8966-4d39-9073-c2e3ffc369fc None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:01.103620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da54a15-8966-4d39-9073-c2e3ffc369fc None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:01.139381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c6abda-b03a-46ea-8a2a-f0f88e6bb97b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:01.140921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c6abda-b03a-46ea-8a2a-f0f88e6bb97b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:01.141436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c6abda-b03a-46ea-8a2a-f0f88e6bb97b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:01.141999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c6abda-b03a-46ea-8a2a-f0f88e6bb97b None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:01.143015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c6abda-b03a-46ea-8a2a-f0f88e6bb97b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:01.143530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c6abda-b03a-46ea-8a2a-f0f88e6bb97b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:01.145238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c6abda-b03a-46ea-8a2a-f0f88e6bb97b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:01.166981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 427/867] 10.222.0.40 () {56 vars in 1149 bytes} [Fri Jul 29 04:35:00 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:01.173757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2a033c-0e22-43fc-94a4-5b540d142fb5 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:01.278556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2a033c-0e22-43fc-94a4-5b540d142fb5 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:01.313895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beccf97-77e4-48aa-b369-b84d562e8298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:01.314519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beccf97-77e4-48aa-b369-b84d562e8298 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:01.314802 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beccf97-77e4-48aa-b369-b84d562e8298 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:01.315082 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6beccf97-77e4-48aa-b369-b84d562e8298 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:01.357972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:01.382977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beccf97-77e4-48aa-b369-b84d562e8298 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:01.384490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6beccf97-77e4-48aa-b369-b84d562e8298 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:01.487444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 441/868] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:01 2022] GET /identity/v3/auth/tokens => generated 1252 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:35:01.493090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c6abda-b03a-46ea-8a2a-f0f88e6bb97b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:01.530338 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:01.602301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c6abda-b03a-46ea-8a2a-f0f88e6bb97b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:01.637787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cadb12-9a96-4980-b707-e0ec2dab41c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:01.638355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cadb12-9a96-4980-b707-e0ec2dab41c0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:01.638635 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cadb12-9a96-4980-b707-e0ec2dab41c0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:01.638907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cadb12-9a96-4980-b707-e0ec2dab41c0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:01.654047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cadb12-9a96-4980-b707-e0ec2dab41c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:01.655510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cadb12-9a96-4980-b707-e0ec2dab41c0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-a9cadb12-9a96-4980-b707-e0ec2dab41c0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:01.674860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:01.675765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:01.676667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:01.676667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:01.676667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:01.676667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:01.679651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 428/869] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:01.682114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beccf97-77e4-48aa-b369-b84d562e8298 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:01.759338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:01.785443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6beccf97-77e4-48aa-b369-b84d562e8298 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:01.820025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29c2aae-aedf-4afd-aedd-c1f0cb6850e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:01.820673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29c2aae-aedf-4afd-aedd-c1f0cb6850e4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:01.820959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29c2aae-aedf-4afd-aedd-c1f0cb6850e4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:01.821234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29c2aae-aedf-4afd-aedd-c1f0cb6850e4 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:01.821978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29c2aae-aedf-4afd-aedd-c1f0cb6850e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:01.822291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29c2aae-aedf-4afd-aedd-c1f0cb6850e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:01.823671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29c2aae-aedf-4afd-aedd-c1f0cb6850e4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:01.840696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 442/870] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:01.842862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cadb12-9a96-4980-b707-e0ec2dab41c0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:01.916655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:01.950932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cadb12-9a96-4980-b707-e0ec2dab41c0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:01.989296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18230baf-9233-47d5-ba53-eb2ef841799c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:01.989957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18230baf-9233-47d5-ba53-eb2ef841799c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:01.990240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18230baf-9233-47d5-ba53-eb2ef841799c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:01.990515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18230baf-9233-47d5-ba53-eb2ef841799c None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:01.998278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18230baf-9233-47d5-ba53-eb2ef841799c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:01.999752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18230baf-9233-47d5-ba53-eb2ef841799c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-18230baf-9233-47d5-ba53-eb2ef841799c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:35:02.012507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:02.013462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:02.014335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:35:02.014335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:35:02.014335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:35:02.014335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:35:02.014335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:35:02.014335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:35:02.014335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:02.014335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:02.019019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 429/871] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:02.020374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29c2aae-aedf-4afd-aedd-c1f0cb6850e4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:02.124502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29c2aae-aedf-4afd-aedd-c1f0cb6850e4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:02.128219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b7d13c-0c58-44eb-8145-9bea4655fab8 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.128510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b7d13c-0c58-44eb-8145-9bea4655fab8 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.128809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b7d13c-0c58-44eb-8145-9bea4655fab8 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.130511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 430/872] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:35:02.138105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eab7bd6-191c-41c2-aee8-19c571df4e2e None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.138514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eab7bd6-191c-41c2-aee8-19c571df4e2e None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.138977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eab7bd6-191c-41c2-aee8-19c571df4e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.159414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-0eab7bd6-191c-41c2-aee8-19c571df4e2e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:02.160885 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0eab7bd6-191c-41c2-aee8-19c571df4e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:02.162792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a7fa50-e794-41c6-82fd-664240446ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:02.163419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a7fa50-e794-41c6-82fd-664240446ca5 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.164359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a7fa50-e794-41c6-82fd-664240446ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.164359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a7fa50-e794-41c6-82fd-664240446ca5 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.164754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a7fa50-e794-41c6-82fd-664240446ca5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:02.165069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a7fa50-e794-41c6-82fd-664240446ca5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:02.166507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a7fa50-e794-41c6-82fd-664240446ca5 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:02.168295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:02.184098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 443/873] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:35:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:02.184098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f28d55-6ada-49bb-8c0e-e4a25e68214d None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.184098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f28d55-6ada-49bb-8c0e-e4a25e68214d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.184098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f28d55-6ada-49bb-8c0e-e4a25e68214d None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.184098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 444/874] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:02 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:02.187797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f28d55-6ada-49bb-8c0e-e4a25e68214d None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:02.248622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 431/875] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:02 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:02.248622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e151b3-bbdb-4290-ae08-2dbd7ab67e56 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.248622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e151b3-bbdb-4290-ae08-2dbd7ab67e56 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.248622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e151b3-bbdb-4290-ae08-2dbd7ab67e56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.270002 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f4e151b3-bbdb-4290-ae08-2dbd7ab67e56 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:02.271212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f4e151b3-bbdb-4290-ae08-2dbd7ab67e56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:02.299910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f28d55-6ada-49bb-8c0e-e4a25e68214d None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:02.331247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:02.335686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3e1284-9586-4b06-b877-d0f145d1d6ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:02.336402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3e1284-9586-4b06-b877-d0f145d1d6ad None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.336783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3e1284-9586-4b06-b877-d0f145d1d6ad None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.337074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3e1284-9586-4b06-b877-d0f145d1d6ad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.337873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3e1284-9586-4b06-b877-d0f145d1d6ad None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=3f53019d35f244e4959d8340ba6f7e45) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:02.338222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3e1284-9586-4b06-b877-d0f145d1d6ad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:02.339647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3e1284-9586-4b06-b877-d0f145d1d6ad None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:02.344646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 432/876] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:02 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:02.345690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e151b3-bbdb-4290-ae08-2dbd7ab67e56 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:02.356628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 445/877] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:02 2022] GET /identity/v3/role_assignments?user.id=3f53019d35f244e4959d8340ba6f7e45&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:02.367380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3e1284-9586-4b06-b877-d0f145d1d6ad None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:02.437744 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:02.477975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e151b3-bbdb-4290-ae08-2dbd7ab67e56 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:02.487467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3e1284-9586-4b06-b877-d0f145d1d6ad None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:02.519644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9268e754-674e-4bf1-a9f2-dcf940045712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:02.520401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268e754-674e-4bf1-a9f2-dcf940045712 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.520776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268e754-674e-4bf1-a9f2-dcf940045712 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.521067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268e754-674e-4bf1-a9f2-dcf940045712 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.521749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9268e754-674e-4bf1-a9f2-dcf940045712 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:02.523304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9268e754-674e-4bf1-a9f2-dcf940045712 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:02.526099 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f52a1a-a11c-499a-a341-0f98d854e0d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:02.526982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f52a1a-a11c-499a-a341-0f98d854e0d9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.527463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f52a1a-a11c-499a-a341-0f98d854e0d9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.527748 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f52a1a-a11c-499a-a341-0f98d854e0d9 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.528390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f52a1a-a11c-499a-a341-0f98d854e0d9 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:02.529804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f52a1a-a11c-499a-a341-0f98d854e0d9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-9268e754-674e-4bf1-a9f2-dcf940045712 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:02.533692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:02.534662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:02.535688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 433/878] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:35:02 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-b3f52a1a-a11c-499a-a341-0f98d854e0d9 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:02.540730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:02.541699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:02.542506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:02.544695 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9268e754-674e-4bf1-a9f2-dcf940045712 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:02.547473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 446/879] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:02 2022] GET /identity/v3/services/object-store => generated 93 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:02.548911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f52a1a-a11c-499a-a341-0f98d854e0d9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:02.615626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:02.683003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9268e754-674e-4bf1-a9f2-dcf940045712 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:02.694926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f52a1a-a11c-499a-a341-0f98d854e0d9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:02.721366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e215114c-3fc7-432c-9b32-4b356f916c8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:02.721812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e215114c-3fc7-432c-9b32-4b356f916c8c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.722101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e215114c-3fc7-432c-9b32-4b356f916c8c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.722385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e215114c-3fc7-432c-9b32-4b356f916c8c None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.723076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e215114c-3fc7-432c-9b32-4b356f916c8c None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:02.723391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e215114c-3fc7-432c-9b32-4b356f916c8c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:02.724793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e215114c-3fc7-432c-9b32-4b356f916c8c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:02.729567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:02.730065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 434/880] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:35:02 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:02.735848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e215114c-3fc7-432c-9b32-4b356f916c8c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:02.746122 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82a9ba8-75e2-41e6-979b-ddb6217f934e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:02.746758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82a9ba8-75e2-41e6-979b-ddb6217f934e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.747046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82a9ba8-75e2-41e6-979b-ddb6217f934e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.747886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b82a9ba8-75e2-41e6-979b-ddb6217f934e None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.748620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82a9ba8-75e2-41e6-979b-ddb6217f934e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:02.748943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82a9ba8-75e2-41e6-979b-ddb6217f934e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:02.751114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b82a9ba8-75e2-41e6-979b-ddb6217f934e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:02.757965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 447/881] 10.222.0.40 () {56 vars in 1122 bytes} [Fri Jul 29 04:35:02 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:02.763873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82a9ba8-75e2-41e6-979b-ddb6217f934e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:02.872148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e215114c-3fc7-432c-9b32-4b356f916c8c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:02.890635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b82a9ba8-75e2-41e6-979b-ddb6217f934e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:02.891744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:02.911517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5f1a2b-7909-4b7b-a3fd-d3cd1abc7556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:02.912186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5f1a2b-7909-4b7b-a3fd-d3cd1abc7556 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.912476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5f1a2b-7909-4b7b-a3fd-d3cd1abc7556 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.912787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5f1a2b-7909-4b7b-a3fd-d3cd1abc7556 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.913493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5f1a2b-7909-4b7b-a3fd-d3cd1abc7556 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:02.914167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5f1a2b-7909-4b7b-a3fd-d3cd1abc7556 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:02.915286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5f1a2b-7909-4b7b-a3fd-d3cd1abc7556 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:02.924681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 435/882] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:35:02 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:02.926160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b9b26f-1a73-4e72-a009-04d75654c4c5 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.926688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b9b26f-1a73-4e72-a009-04d75654c4c5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.926976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b9b26f-1a73-4e72-a009-04d75654c4c5 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.928445 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 436/883] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:02.928992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b9b26f-1a73-4e72-a009-04d75654c4c5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:02.930520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a8aa73-2948-4705-97df-27334087116e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:02.931202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8aa73-2948-4705-97df-27334087116e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.931489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8aa73-2948-4705-97df-27334087116e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.931826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a8aa73-2948-4705-97df-27334087116e None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.932593 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8aa73-2948-4705-97df-27334087116e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:02.933023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8aa73-2948-4705-97df-27334087116e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:02.934418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a8aa73-2948-4705-97df-27334087116e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:02.939790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 448/884] 10.222.0.40 () {56 vars in 1122 bytes} [Fri Jul 29 04:35:02 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:02.942696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb84631-6a6a-48c5-b0a0-043fb441e428 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:02.942696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb84631-6a6a-48c5-b0a0-043fb441e428 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:02.942696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb84631-6a6a-48c5-b0a0-043fb441e428 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:02.962174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-ceb84631-6a6a-48c5-b0a0-043fb441e428 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:02.963317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ceb84631-6a6a-48c5-b0a0-043fb441e428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:03.041003 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:03.067052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 449/885] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:02 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:03.075398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b9b26f-1a73-4e72-a009-04d75654c4c5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:03.091850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb84631-6a6a-48c5-b0a0-043fb441e428 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:03.121181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d56dd48-503b-420b-b254-678c93fcbfb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:03.122147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d56dd48-503b-420b-b254-678c93fcbfb4 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:03.122536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d56dd48-503b-420b-b254-678c93fcbfb4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:03.122815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d56dd48-503b-420b-b254-678c93fcbfb4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:03.124245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d56dd48-503b-420b-b254-678c93fcbfb4 None admin] RBAC: Adding query filter params (interface=public, service_id=c5746fe2b75541d3927445a89c347d96, region_id=RegionOne) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:03.124563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d56dd48-503b-420b-b254-678c93fcbfb4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:03.126236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d56dd48-503b-420b-b254-678c93fcbfb4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:03.134665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 437/886] 10.222.0.40 () {56 vars in 1251 bytes} [Fri Jul 29 04:35:02 2022] GET /identity/v3/endpoints?service_id=c5746fe2b75541d3927445a89c347d96&interface=public®ion_id=RegionOne => generated 194 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:03.140226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d56dd48-503b-420b-b254-678c93fcbfb4 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:03.213062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:03.225220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb84631-6a6a-48c5-b0a0-043fb441e428 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:03.263966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-607c9fcf-c9bc-4b76-9f08-e7abd83997cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:03.264619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607c9fcf-c9bc-4b76-9f08-e7abd83997cd None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:03.264907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607c9fcf-c9bc-4b76-9f08-e7abd83997cd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:03.265247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607c9fcf-c9bc-4b76-9f08-e7abd83997cd None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:03.266568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607c9fcf-c9bc-4b76-9f08-e7abd83997cd None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:03.267967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-607c9fcf-c9bc-4b76-9f08-e7abd83997cd None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:03.273565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d56dd48-503b-420b-b254-678c93fcbfb4 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:03.291800 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 450/887] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:35:03 2022] POST /identity/v3/services => generated 269 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:35:03.325545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ae8680-c2bb-4444-a54a-08b6b5de9791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:03.326186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae8680-c2bb-4444-a54a-08b6b5de9791 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:03.326534 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae8680-c2bb-4444-a54a-08b6b5de9791 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:03.326813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ae8680-c2bb-4444-a54a-08b6b5de9791 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:03.327477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ae8680-c2bb-4444-a54a-08b6b5de9791 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:03.330248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ae8680-c2bb-4444-a54a-08b6b5de9791 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:03.337502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 438/888] 10.222.0.40 () {56 vars in 1087 bytes} [Fri Jul 29 04:35:03 2022] GET /identity/v3/services => generated 1268 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:35:04.523133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f100f225-3924-465c-92dd-7a25272a02fa None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:04.523686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f100f225-3924-465c-92dd-7a25272a02fa None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:04.523686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f100f225-3924-465c-92dd-7a25272a02fa None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:04.525445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 451/889] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:04.530711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b5d851-a8d9-419b-946d-8580bc715b01 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:04.531767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b5d851-a8d9-419b-946d-8580bc715b01 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:04.531846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b5d851-a8d9-419b-946d-8580bc715b01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:04.550373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-b5b5d851-a8d9-419b-946d-8580bc715b01 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:04.551582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b5b5d851-a8d9-419b-946d-8580bc715b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:04.624382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 439/890] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:04 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:04.648439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f100f225-3924-465c-92dd-7a25272a02fa None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:04.695522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:04.758899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f100f225-3924-465c-92dd-7a25272a02fa None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:04.794506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e7169f-b7c6-40bb-8541-cc3c1519465b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:04.795079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e7169f-b7c6-40bb-8541-cc3c1519465b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:04.795369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e7169f-b7c6-40bb-8541-cc3c1519465b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:04.795652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e7169f-b7c6-40bb-8541-cc3c1519465b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:04.806859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e7169f-b7c6-40bb-8541-cc3c1519465b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c7e7169f-b7c6-40bb-8541-cc3c1519465b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:04.812999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:04.814116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:04.815220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 452/891] 10.222.0.40 () {56 vars in 1120 bytes} [Fri Jul 29 04:35:04 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:04.820575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b5d851-a8d9-419b-946d-8580bc715b01 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:04.857834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:04.927279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b5d851-a8d9-419b-946d-8580bc715b01 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:04.962968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce15359f-f6e8-44fa-b208-9523bf2fe2a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:04.963600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce15359f-f6e8-44fa-b208-9523bf2fe2a2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:04.963886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce15359f-f6e8-44fa-b208-9523bf2fe2a2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:04.964168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce15359f-f6e8-44fa-b208-9523bf2fe2a2 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:04.964931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce15359f-f6e8-44fa-b208-9523bf2fe2a2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:04.965241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce15359f-f6e8-44fa-b208-9523bf2fe2a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:04.966615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce15359f-f6e8-44fa-b208-9523bf2fe2a2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:04.975302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 440/892] 10.222.0.40 () {56 vars in 1115 bytes} [Fri Jul 29 04:35:04 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:04.980867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e7169f-b7c6-40bb-8541-cc3c1519465b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:05.087102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e7169f-b7c6-40bb-8541-cc3c1519465b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:05.088754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:05.124102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebe83a4-9dc9-40d4-8b92-3880a7ab77d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:05.124844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebe83a4-9dc9-40d4-8b92-3880a7ab77d1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.125131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebe83a4-9dc9-40d4-8b92-3880a7ab77d1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.125410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebe83a4-9dc9-40d4-8b92-3880a7ab77d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.135811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebe83a4-9dc9-40d4-8b92-3880a7ab77d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:05.137313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebe83a4-9dc9-40d4-8b92-3880a7ab77d1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:05.149067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506afc9a-b002-4487-8174-3d7568b04a57 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.149399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506afc9a-b002-4487-8174-3d7568b04a57 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.149643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506afc9a-b002-4487-8174-3d7568b04a57 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.151273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 441/893] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-0ebe83a4-9dc9-40d4-8b92-3880a7ab77d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:05.154031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:05.155155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:05.156298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:05.156298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:05.156298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:05.156298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:05.156298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 453/894] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:05.157030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38370b6a-d316-4a49-95e5-742543aa22dd None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.157030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38370b6a-d316-4a49-95e5-742543aa22dd None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.157030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38370b6a-d316-4a49-95e5-742543aa22dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.160999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebe83a4-9dc9-40d4-8b92-3880a7ab77d1 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:05.176729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-38370b6a-d316-4a49-95e5-742543aa22dd None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:05.177974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-38370b6a-d316-4a49-95e5-742543aa22dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:05.266944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 442/895] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:05 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:05.283759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebe83a4-9dc9-40d4-8b92-3880a7ab77d1 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:05.289227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38370b6a-d316-4a49-95e5-742543aa22dd None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:05.323827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:05.337554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6de749-ca55-4ac4-9c3a-656d31be3c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:05.338536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6de749-ca55-4ac4-9c3a-656d31be3c33 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.339083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6de749-ca55-4ac4-9c3a-656d31be3c33 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.339571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6de749-ca55-4ac4-9c3a-656d31be3c33 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.340922 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6de749-ca55-4ac4-9c3a-656d31be3c33 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:05.341171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6de749-ca55-4ac4-9c3a-656d31be3c33 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:05.345849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6de749-ca55-4ac4-9c3a-656d31be3c33 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:05.364356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 454/896] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:05.369372 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6de749-ca55-4ac4-9c3a-656d31be3c33 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:05.385511 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:05.421094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38370b6a-d316-4a49-95e5-742543aa22dd None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:05.457891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07ccebe-ec70-494f-89e5-4f5d7745dd8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:05.458522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07ccebe-ec70-494f-89e5-4f5d7745dd8d None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.458819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07ccebe-ec70-494f-89e5-4f5d7745dd8d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.459106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07ccebe-ec70-494f-89e5-4f5d7745dd8d None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.459821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07ccebe-ec70-494f-89e5-4f5d7745dd8d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:05.461292 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07ccebe-ec70-494f-89e5-4f5d7745dd8d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:05.477737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 443/897] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:35:05 2022] POST /identity/v3/services => generated 241 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:35:05.478545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd32224-e21f-4872-8e26-07f298253a2d None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.478833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd32224-e21f-4872-8e26-07f298253a2d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.479115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd32224-e21f-4872-8e26-07f298253a2d None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.480551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 444/898] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:05.485023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6de749-ca55-4ac4-9c3a-656d31be3c33 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:05.486565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5c8343-1b36-41d9-8819-15a6e442359f None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.486862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5c8343-1b36-41d9-8819-15a6e442359f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.487200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5c8343-1b36-41d9-8819-15a6e442359f None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.489866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 445/899] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:05 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:05.491330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f429466e-3448-4a78-99da-a4fec6ac6a48 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.491625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f429466e-3448-4a78-99da-a4fec6ac6a48 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.491907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f429466e-3448-4a78-99da-a4fec6ac6a48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.513834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f429466e-3448-4a78-99da-a4fec6ac6a48 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:05.515605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f429466e-3448-4a78-99da-a4fec6ac6a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:05.541016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c1fb483-b857-4cd8-8387-408c51dddca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:05.542547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1fb483-b857-4cd8-8387-408c51dddca4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.543242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1fb483-b857-4cd8-8387-408c51dddca4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.543600 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c1fb483-b857-4cd8-8387-408c51dddca4 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.554576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1fb483-b857-4cd8-8387-408c51dddca4 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:05.556403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c1fb483-b857-4cd8-8387-408c51dddca4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-2c1fb483-b857-4cd8-8387-408c51dddca4 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:35:05.572673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:05.573732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:05.574656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 455/900] 10.222.0.40 () {56 vars in 1117 bytes} [Fri Jul 29 04:35:05 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:05.576466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8471481-d60c-4317-9e4e-5b2719f9a623 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.576466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8471481-d60c-4317-9e4e-5b2719f9a623 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.576466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8471481-d60c-4317-9e4e-5b2719f9a623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.600916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c8471481-d60c-4317-9e4e-5b2719f9a623 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:05.602635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c8471481-d60c-4317-9e4e-5b2719f9a623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:05.608851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 446/901] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:05 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:05.608851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f429466e-3448-4a78-99da-a4fec6ac6a48 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:05.637745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:05.702009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:05.708640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 456/902] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:05 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:05.709443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8471481-d60c-4317-9e4e-5b2719f9a623 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:05.747860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f429466e-3448-4a78-99da-a4fec6ac6a48 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:05.786840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f0367c-e7b9-42e6-a1ed-375a6c8fef88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:05.787535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f0367c-e7b9-42e6-a1ed-375a6c8fef88 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.787829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f0367c-e7b9-42e6-a1ed-375a6c8fef88 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.788108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f0367c-e7b9-42e6-a1ed-375a6c8fef88 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.789012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f0367c-e7b9-42e6-a1ed-375a6c8fef88 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:05.789328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f0367c-e7b9-42e6-a1ed-375a6c8fef88 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:05.790817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f0367c-e7b9-42e6-a1ed-375a6c8fef88 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:05.812062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 447/903] 10.222.0.40 () {56 vars in 1149 bytes} [Fri Jul 29 04:35:05 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:05.813030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f0367c-e7b9-42e6-a1ed-375a6c8fef88 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:05.849071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8471481-d60c-4317-9e4e-5b2719f9a623 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:05.895767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fa758b-596d-4b57-be6d-7d1c8c716812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:05.897170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa758b-596d-4b57-be6d-7d1c8c716812 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.897648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa758b-596d-4b57-be6d-7d1c8c716812 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.898116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fa758b-596d-4b57-be6d-7d1c8c716812 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.899218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fa758b-596d-4b57-be6d-7d1c8c716812 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:05.901681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fa758b-596d-4b57-be6d-7d1c8c716812 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-a1fa758b-596d-4b57-be6d-7d1c8c716812 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:05.913021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:05.914000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:05.915067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 457/904] 10.222.0.40 () {56 vars in 1132 bytes} [Fri Jul 29 04:35:05 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:05.916957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fa758b-596d-4b57-be6d-7d1c8c716812 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:05.918331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:05.947101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f0367c-e7b9-42e6-a1ed-375a6c8fef88 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:05.986474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96e475e-bf61-40d6-b7f9-fa0ba1d716b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:05.987294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96e475e-bf61-40d6-b7f9-fa0ba1d716b3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:05.987588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96e475e-bf61-40d6-b7f9-fa0ba1d716b3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:05.988234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96e475e-bf61-40d6-b7f9-fa0ba1d716b3 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:05.989577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96e475e-bf61-40d6-b7f9-fa0ba1d716b3 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:05.991718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96e475e-bf61-40d6-b7f9-fa0ba1d716b3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:05.997226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-e96e475e-bf61-40d6-b7f9-fa0ba1d716b3 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:06.009115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:06.010149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 29 04:35:06.010949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:06.016794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 448/905] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:35:05 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:06.017716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96e475e-bf61-40d6-b7f9-fa0ba1d716b3 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:06.049451 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fa758b-596d-4b57-be6d-7d1c8c716812 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:06.091153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0c0204-f64f-40ba-9339-f3f103c8b32c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:06.091814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c0204-f64f-40ba-9339-f3f103c8b32c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:06.092086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c0204-f64f-40ba-9339-f3f103c8b32c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:06.092343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0c0204-f64f-40ba-9339-f3f103c8b32c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:06.144801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96e475e-bf61-40d6-b7f9-fa0ba1d716b3 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:06.178055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0c0204-f64f-40ba-9339-f3f103c8b32c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:06.179526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0c0204-f64f-40ba-9339-f3f103c8b32c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:06.184151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9206d8c7-368e-420a-ac68-51ac4d84f089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:06.184816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9206d8c7-368e-420a-ac68-51ac4d84f089 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:06.185109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9206d8c7-368e-420a-ac68-51ac4d84f089 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:06.185392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9206d8c7-368e-420a-ac68-51ac4d84f089 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:06.186082 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9206d8c7-368e-420a-ac68-51ac4d84f089 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:06.186398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9206d8c7-368e-420a-ac68-51ac4d84f089 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:06.188943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9206d8c7-368e-420a-ac68-51ac4d84f089 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:06.198875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 449/906] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:06 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:06.199745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9206d8c7-368e-420a-ac68-51ac4d84f089 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:06.204857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:06.263916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:06.307750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9206d8c7-368e-420a-ac68-51ac4d84f089 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:06.318737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 458/907] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:05 2022] GET /identity/v3/auth/tokens => generated 1464 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:35:06.318737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0c0204-f64f-40ba-9339-f3f103c8b32c None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:06.348243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78709ed3-1e66-4189-ab39-d6f638c502b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:06.348899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78709ed3-1e66-4189-ab39-d6f638c502b7 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:06.349187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78709ed3-1e66-4189-ab39-d6f638c502b7 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:06.349475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78709ed3-1e66-4189-ab39-d6f638c502b7 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:06.350159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78709ed3-1e66-4189-ab39-d6f638c502b7 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:06.350472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78709ed3-1e66-4189-ab39-d6f638c502b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:06.352642 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78709ed3-1e66-4189-ab39-d6f638c502b7 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:06.361657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 450/908] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:35:06 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:06.361657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78709ed3-1e66-4189-ab39-d6f638c502b7 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:06.429477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:06.438129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0c0204-f64f-40ba-9339-f3f103c8b32c None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:06.467014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78709ed3-1e66-4189-ab39-d6f638c502b7 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:06.473908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ded7118-c7df-4031-bdd6-03bfa30c1fcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:06.474650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded7118-c7df-4031-bdd6-03bfa30c1fcc None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:06.474940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded7118-c7df-4031-bdd6-03bfa30c1fcc None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:06.475219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded7118-c7df-4031-bdd6-03bfa30c1fcc None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:06.475902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ded7118-c7df-4031-bdd6-03bfa30c1fcc None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:06.476329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ded7118-c7df-4031-bdd6-03bfa30c1fcc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:06.477759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ded7118-c7df-4031-bdd6-03bfa30c1fcc None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:06.487907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 459/909] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:06 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:06.487907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ded7118-c7df-4031-bdd6-03bfa30c1fcc None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:06.507346 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519981f-f222-4257-8cbd-8f5465b57699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:06.508003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519981f-f222-4257-8cbd-8f5465b57699 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:06.508291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519981f-f222-4257-8cbd-8f5465b57699 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:06.508570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519981f-f222-4257-8cbd-8f5465b57699 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:06.517328 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:06.520855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519981f-f222-4257-8cbd-8f5465b57699 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:06.522328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519981f-f222-4257-8cbd-8f5465b57699 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-7519981f-f222-4257-8cbd-8f5465b57699 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:06.536866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:06.537782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:06.538677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:06.538677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:06.538677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:06.538677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:06.538677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 451/910] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:06.538959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519981f-f222-4257-8cbd-8f5465b57699 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:06.600058 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ded7118-c7df-4031-bdd6-03bfa30c1fcc None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:06.637887 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13391413-8df8-4850-baa2-d19718b09d10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:06.638520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13391413-8df8-4850-baa2-d19718b09d10 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:06.638805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13391413-8df8-4850-baa2-d19718b09d10 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:06.639090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13391413-8df8-4850-baa2-d19718b09d10 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:06.639794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13391413-8df8-4850-baa2-d19718b09d10 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:06.640108 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13391413-8df8-4850-baa2-d19718b09d10 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:06.641798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13391413-8df8-4850-baa2-d19718b09d10 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:06.644143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519981f-f222-4257-8cbd-8f5465b57699 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:06.646705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 460/911] 10.222.0.40 () {56 vars in 1114 bytes} [Fri Jul 29 04:35:06 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:06.647513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13391413-8df8-4850-baa2-d19718b09d10 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:06.677286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:06.693693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18155fc-b20e-4feb-9195-bcbbad98d0a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:06.694254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18155fc-b20e-4feb-9195-bcbbad98d0a1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:06.694543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18155fc-b20e-4feb-9195-bcbbad98d0a1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:06.694821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c18155fc-b20e-4feb-9195-bcbbad98d0a1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:06.695582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18155fc-b20e-4feb-9195-bcbbad98d0a1 None admin] RBAC: Adding query filter params (interface=public, service_id=cef8157dd63a42b699058be979838715, region_id=RegionOne) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:06.695906 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18155fc-b20e-4feb-9195-bcbbad98d0a1 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:06.697310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c18155fc-b20e-4feb-9195-bcbbad98d0a1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:06.706941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 452/912] 10.222.0.40 () {56 vars in 1251 bytes} [Fri Jul 29 04:35:06 2022] GET /identity/v3/endpoints?service_id=cef8157dd63a42b699058be979838715&interface=public®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:06.706941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18155fc-b20e-4feb-9195-bcbbad98d0a1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:06.757160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:06.768497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13391413-8df8-4850-baa2-d19718b09d10 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:06.805273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf8e744-9ccc-4544-ba41-524a45d5e262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:06.806238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf8e744-9ccc-4544-ba41-524a45d5e262 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:06.806684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf8e744-9ccc-4544-ba41-524a45d5e262 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:06.806970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf8e744-9ccc-4544-ba41-524a45d5e262 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:06.807668 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf8e744-9ccc-4544-ba41-524a45d5e262 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:06.807982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf8e744-9ccc-4544-ba41-524a45d5e262 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:06.809440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf8e744-9ccc-4544-ba41-524a45d5e262 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:06.814560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c18155fc-b20e-4feb-9195-bcbbad98d0a1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:06.825638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 461/913] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:06.825638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf8e744-9ccc-4544-ba41-524a45d5e262 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:06.856464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c04d3db-4391-41ad-be0e-c16996d2b0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:06.857151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c04d3db-4391-41ad-be0e-c16996d2b0c3 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:06.857434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c04d3db-4391-41ad-be0e-c16996d2b0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:06.857720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c04d3db-4391-41ad-be0e-c16996d2b0c3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:06.858429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c04d3db-4391-41ad-be0e-c16996d2b0c3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:06.860229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c04d3db-4391-41ad-be0e-c16996d2b0c3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:06.881883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 453/914] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:35:06 2022] POST /identity/v3/endpoints => generated 346 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:35:06.883062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c04d3db-4391-41ad-be0e-c16996d2b0c3 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:06.946090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:06.954496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf8e744-9ccc-4544-ba41-524a45d5e262 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:07.008231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb39ed5-3ca6-4c02-8058-dd05c58f5e76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:07.008903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb39ed5-3ca6-4c02-8058-dd05c58f5e76 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:07.009194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb39ed5-3ca6-4c02-8058-dd05c58f5e76 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:07.009476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb39ed5-3ca6-4c02-8058-dd05c58f5e76 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:07.010144 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb39ed5-3ca6-4c02-8058-dd05c58f5e76 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:07.011554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb39ed5-3ca6-4c02-8058-dd05c58f5e76 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:07.020259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 462/915] 10.222.0.40 () {56 vars in 1087 bytes} [Fri Jul 29 04:35:06 2022] GET /identity/v3/services => generated 1497 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:35:07.041008 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c04d3db-4391-41ad-be0e-c16996d2b0c3 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:07.083994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c424d6e2-369b-4fef-ba3c-18b07cfc8e1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:07.084954 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c424d6e2-369b-4fef-ba3c-18b07cfc8e1b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:07.085431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c424d6e2-369b-4fef-ba3c-18b07cfc8e1b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:07.085713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c424d6e2-369b-4fef-ba3c-18b07cfc8e1b None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:07.094173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c424d6e2-369b-4fef-ba3c-18b07cfc8e1b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:07.095672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c424d6e2-369b-4fef-ba3c-18b07cfc8e1b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-c424d6e2-369b-4fef-ba3c-18b07cfc8e1b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:35:07.108859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:07.110732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:07.111676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:35:07.111676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:35:07.111676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:35:07.111676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:35:07.111676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:35:07.111676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:35:07.111676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:07.111676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:07.111676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 454/916] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:07.117188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb39ed5-3ca6-4c02-8058-dd05c58f5e76 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:07.132782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:07.236698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb39ed5-3ca6-4c02-8058-dd05c58f5e76 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:07.273723 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa460ef-b6f9-4d96-b518-8737a31d11d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:07.276848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa460ef-b6f9-4d96-b518-8737a31d11d2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:07.276848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa460ef-b6f9-4d96-b518-8737a31d11d2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:07.276848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa460ef-b6f9-4d96-b518-8737a31d11d2 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:07.276848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa460ef-b6f9-4d96-b518-8737a31d11d2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:07.277833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa460ef-b6f9-4d96-b518-8737a31d11d2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:07.278751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa460ef-b6f9-4d96-b518-8737a31d11d2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:07.293952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 463/917] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:35:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:07.300481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c424d6e2-369b-4fef-ba3c-18b07cfc8e1b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:07.408617 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:07.412147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c424d6e2-369b-4fef-ba3c-18b07cfc8e1b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:07.448770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea26b1-1871-4cf5-a77e-88c8f4a13076 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:07.449393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea26b1-1871-4cf5-a77e-88c8f4a13076 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:07.449676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea26b1-1871-4cf5-a77e-88c8f4a13076 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:07.449953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea26b1-1871-4cf5-a77e-88c8f4a13076 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:07.450850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fea26b1-1871-4cf5-a77e-88c8f4a13076 None admin] RBAC: Adding query filter params (role.id=8d6877d388f74e08b7653a068705e759, scope.project.id=441bcfe715d44efea69d556677094857, user.id=3f53019d35f244e4959d8340ba6f7e45) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:07.451239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fea26b1-1871-4cf5-a77e-88c8f4a13076 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:07.453389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fea26b1-1871-4cf5-a77e-88c8f4a13076 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:07.459515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 455/918] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:07 2022] GET /identity/v3/role_assignments?user.id=3f53019d35f244e4959d8340ba6f7e45&scope.project.id=441bcfe715d44efea69d556677094857&role.id=8d6877d388f74e08b7653a068705e759 => generated 259 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:07.579425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8266b-8f32-4ce2-92ac-e0a376c551ed None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:07.579613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8266b-8f32-4ce2-92ac-e0a376c551ed None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:07.579900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb8266b-8f32-4ce2-92ac-e0a376c551ed None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:07.581612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 464/919] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:07.587283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addec0f5-33a4-4707-a8f6-9dbfb21b2101 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:07.587601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addec0f5-33a4-4707-a8f6-9dbfb21b2101 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:07.587853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addec0f5-33a4-4707-a8f6-9dbfb21b2101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:07.607858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-addec0f5-33a4-4707-a8f6-9dbfb21b2101 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:07.609175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-addec0f5-33a4-4707-a8f6-9dbfb21b2101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:07.688732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 456/920] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:07 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:07.710774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb8266b-8f32-4ce2-92ac-e0a376c551ed None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:07.791438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:07.828309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb8266b-8f32-4ce2-92ac-e0a376c551ed None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:07.865643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6342256a-48b0-400e-8fb0-a30497701478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:07.866658 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6342256a-48b0-400e-8fb0-a30497701478 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:07.867153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6342256a-48b0-400e-8fb0-a30497701478 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:07.867645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6342256a-48b0-400e-8fb0-a30497701478 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:07.868478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6342256a-48b0-400e-8fb0-a30497701478 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:07.870085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6342256a-48b0-400e-8fb0-a30497701478 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-6342256a-48b0-400e-8fb0-a30497701478 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:07.880365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:07.881464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:07.882368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:07.885222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 465/921] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:07 2022] GET /identity/v3/services/object-store => generated 93 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:07.896623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addec0f5-33a4-4707-a8f6-9dbfb21b2101 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:07.988709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:07.994617 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addec0f5-33a4-4707-a8f6-9dbfb21b2101 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:08.029905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb34c8cf-7045-4c07-a77c-b11807029056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:08.030526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb34c8cf-7045-4c07-a77c-b11807029056 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:08.031530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb34c8cf-7045-4c07-a77c-b11807029056 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:08.032081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb34c8cf-7045-4c07-a77c-b11807029056 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:08.032526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb34c8cf-7045-4c07-a77c-b11807029056 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:08.032863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb34c8cf-7045-4c07-a77c-b11807029056 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:08.034229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb34c8cf-7045-4c07-a77c-b11807029056 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:08.039282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 457/922] 10.222.0.40 () {56 vars in 1122 bytes} [Fri Jul 29 04:35:07 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:08.044686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6342256a-48b0-400e-8fb0-a30497701478 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:08.152318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6342256a-48b0-400e-8fb0-a30497701478 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:08.197091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e529bb-521b-4625-91fb-e836386d68ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:08.197715 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e529bb-521b-4625-91fb-e836386d68ca None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:08.197995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e529bb-521b-4625-91fb-e836386d68ca None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:08.198273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e529bb-521b-4625-91fb-e836386d68ca None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:08.198959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e529bb-521b-4625-91fb-e836386d68ca None admin] RBAC: Adding query filter params (type=object-store) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:08.199274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e529bb-521b-4625-91fb-e836386d68ca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:08.200657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e529bb-521b-4625-91fb-e836386d68ca None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:08.201884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:08.207912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 466/923] 10.222.0.40 () {56 vars in 1122 bytes} [Fri Jul 29 04:35:08 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:08.213254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb34c8cf-7045-4c07-a77c-b11807029056 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:08.320782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb34c8cf-7045-4c07-a77c-b11807029056 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:08.357049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cec28c7-bef9-4ec4-88a0-6e10bc15c38f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:08.357673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cec28c7-bef9-4ec4-88a0-6e10bc15c38f None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:08.357957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cec28c7-bef9-4ec4-88a0-6e10bc15c38f None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:08.358232 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cec28c7-bef9-4ec4-88a0-6e10bc15c38f None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:08.359042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cec28c7-bef9-4ec4-88a0-6e10bc15c38f None admin] RBAC: Adding query filter params (interface=public, service_id=23fba158366349ea9b69cda5251a24fb, region_id=RegionOne) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:08.359042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cec28c7-bef9-4ec4-88a0-6e10bc15c38f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:08.360721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cec28c7-bef9-4ec4-88a0-6e10bc15c38f None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:08.366286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 458/924] 10.222.0.40 () {56 vars in 1251 bytes} [Fri Jul 29 04:35:08 2022] GET /identity/v3/endpoints?service_id=23fba158366349ea9b69cda5251a24fb&interface=public®ion_id=RegionOne => generated 194 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:08.371768 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e529bb-521b-4625-91fb-e836386d68ca None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:08.478054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e529bb-521b-4625-91fb-e836386d68ca None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:08.512027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f358376-e476-474f-be6a-7c99046f1aac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:08.512681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f358376-e476-474f-be6a-7c99046f1aac None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:08.513101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f358376-e476-474f-be6a-7c99046f1aac None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:08.513376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f358376-e476-474f-be6a-7c99046f1aac None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:08.514033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f358376-e476-474f-be6a-7c99046f1aac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:08.515797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f358376-e476-474f-be6a-7c99046f1aac None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:08.522012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 467/925] 10.222.0.40 () {56 vars in 1087 bytes} [Fri Jul 29 04:35:08 2022] GET /identity/v3/services => generated 1497 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:35:09.145285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03323cf4-d269-410e-b0cc-becf1410add9 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.146318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03323cf4-d269-410e-b0cc-becf1410add9 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.146834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03323cf4-d269-410e-b0cc-becf1410add9 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.148718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 459/926] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:09.154345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e3c7a2-71ec-47a5-b987-c570a3cf973c None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.155172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e3c7a2-71ec-47a5-b987-c570a3cf973c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.155773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e3c7a2-71ec-47a5-b987-c570a3cf973c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.175082 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-58e3c7a2-71ec-47a5-b987-c570a3cf973c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:09.176695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-58e3c7a2-71ec-47a5-b987-c570a3cf973c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:09.241962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:09.242319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a6246-decf-4898-92b5-f6329438e834 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.242832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a6246-decf-4898-92b5-f6329438e834 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.243326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841a6246-decf-4898-92b5-f6329438e834 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.246254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 460/927] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:35:09.251371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea09536-28a3-49bc-a5ac-429d1af74faa None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.251938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea09536-28a3-49bc-a5ac-429d1af74faa None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.252456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea09536-28a3-49bc-a5ac-429d1af74faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.253557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 468/928] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:09 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:09.273641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-bea09536-28a3-49bc-a5ac-429d1af74faa None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:09.275194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-bea09536-28a3-49bc-a5ac-429d1af74faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:09.276679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e3c7a2-71ec-47a5-b987-c570a3cf973c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:09.297772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:09.371569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 461/929] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:09 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:09.391399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea09536-28a3-49bc-a5ac-429d1af74faa None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:09.394829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e3c7a2-71ec-47a5-b987-c570a3cf973c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:09.435240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1327e0-7300-42e9-8435-5dc045694b0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:09.435851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1327e0-7300-42e9-8435-5dc045694b0a None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.436144 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1327e0-7300-42e9-8435-5dc045694b0a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.436425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1327e0-7300-42e9-8435-5dc045694b0a None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.437207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1327e0-7300-42e9-8435-5dc045694b0a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:09.438927 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1327e0-7300-42e9-8435-5dc045694b0a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:09.441642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: WARNING keystone.server.flask.common [None req-cd1327e0-7300-42e9-8435-5dc045694b0a None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 29 04:35:09.468962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.sql.core [None req-cd1327e0-7300-42e9-8435-5dc045694b0a None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 29 04:35:09.468962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 29 04:35:09.468962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [parameters: {'id': '8ac406a1ff084efb876886e7a27c8238', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 29 04:35:09.468962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=92189) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-cd1327e0-7300-42e9-8435-5dc045694b0a None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 8ac406a1ff084efb876886e7a27c8238.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 8ac406a1ff084efb876886e7a27c8238. Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application result = self._query(query) Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application conn.query(q) Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 29 04:35:09.494382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application result.read() Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Jul 29 04:35:09.496192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return project_ref.to_dict() Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application next(self.gen) Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application yield resource Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application next(self.gen) Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self.session.rollback() Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self.force_reraise() Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise self.value Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application session.commit() Jul 29 04:35:09.503980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._prepare_impl() Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self.session.flush() Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._flush(objects) Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application compat.raise_( Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application flush_context.execute() Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 29 04:35:09.506190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rec.execute(self) Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 29 04:35:09.508811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application result = self._query(query) Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application conn.query(q) Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application result.read() Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 29 04:35:09.511023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [parameters: {'id': '8ac406a1ff084efb876886e7a27c8238', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:09.512639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:09.513536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.Conflict( Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 8ac406a1ff084efb876886e7a27c8238. Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 469/930] 10.222.0.40 () {60 vars in 1140 bytes} [Fri Jul 29 04:35:09 2022] POST /identity/v3/projects => generated 262 bytes in 222 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dcf5d5-d864-4984-adc1-07394fe72505 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.514450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dcf5d5-d864-4984-adc1-07394fe72505 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.515873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dcf5d5-d864-4984-adc1-07394fe72505 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.515873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 470/931] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:09 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:09.515873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dcf5d5-d864-4984-adc1-07394fe72505 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:09.542178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea09536-28a3-49bc-a5ac-429d1af74faa None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:09.578034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:09.581524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5a650d-82e4-4e76-af2a-fbdaaea93800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:09.582138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5a650d-82e4-4e76-af2a-fbdaaea93800 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.582441 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5a650d-82e4-4e76-af2a-fbdaaea93800 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.582755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5a650d-82e4-4e76-af2a-fbdaaea93800 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.584379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5a650d-82e4-4e76-af2a-fbdaaea93800 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:09.585247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5a650d-82e4-4e76-af2a-fbdaaea93800 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-fc5a650d-82e4-4e76-af2a-fbdaaea93800 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:09.600215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:09.601394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:09.602551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 462/932] 10.222.0.40 () {56 vars in 1132 bytes} [Fri Jul 29 04:35:09 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:09.605164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20471909-58e9-4cbd-8bbc-a1146657217b None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.605443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20471909-58e9-4cbd-8bbc-a1146657217b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.605987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20471909-58e9-4cbd-8bbc-a1146657217b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.634498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-20471909-58e9-4cbd-8bbc-a1146657217b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:09.635733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-20471909-58e9-4cbd-8bbc-a1146657217b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:09.647664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:09.668589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dcf5d5-d864-4984-adc1-07394fe72505 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:09.708417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e14238-86da-4682-8147-36bb285210ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:09.709113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e14238-86da-4682-8147-36bb285210ed None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.709402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e14238-86da-4682-8147-36bb285210ed None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.709915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e14238-86da-4682-8147-36bb285210ed None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.717211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 463/933] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:09 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:09.718090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20471909-58e9-4cbd-8bbc-a1146657217b None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:09.721801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e14238-86da-4682-8147-36bb285210ed None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:09.723507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e14238-86da-4682-8147-36bb285210ed None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-a9e14238-86da-4682-8147-36bb285210ed None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:35:09.746056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:09.747036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:09.747918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:35:09.747918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:35:09.747918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:35:09.747918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:35:09.747918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:35:09.747918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:35:09.747918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:09.747918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:09.747918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 471/934] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:09.748425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e14238-86da-4682-8147-36bb285210ed None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:09.836263 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20471909-58e9-4cbd-8bbc-a1146657217b None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:09.855782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e14238-86da-4682-8147-36bb285210ed None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:09.865716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:09.872487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748ff96e-3f01-4b64-a07d-32ed7aaae4a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:09.873220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748ff96e-3f01-4b64-a07d-32ed7aaae4a7 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.873572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748ff96e-3f01-4b64-a07d-32ed7aaae4a7 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.873973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748ff96e-3f01-4b64-a07d-32ed7aaae4a7 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.875315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748ff96e-3f01-4b64-a07d-32ed7aaae4a7 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:09.875630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748ff96e-3f01-4b64-a07d-32ed7aaae4a7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:09.877045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748ff96e-3f01-4b64-a07d-32ed7aaae4a7 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:09.887115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 464/935] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:09 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:09.887753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748ff96e-3f01-4b64-a07d-32ed7aaae4a7 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:09.893016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f0686c-ffe5-4bed-94f2-24dc608520c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:09.893668 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f0686c-ffe5-4bed-94f2-24dc608520c0 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:09.893957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f0686c-ffe5-4bed-94f2-24dc608520c0 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:09.894238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f0686c-ffe5-4bed-94f2-24dc608520c0 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:09.902168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:09.907201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f0686c-ffe5-4bed-94f2-24dc608520c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-d0f0686c-ffe5-4bed-94f2-24dc608520c0 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:09.914226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:09.915173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:09.916050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 472/936] 10.222.0.40 () {56 vars in 1120 bytes} [Fri Jul 29 04:35:09 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:09.917358 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f0686c-ffe5-4bed-94f2-24dc608520c0 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:09.998309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748ff96e-3f01-4b64-a07d-32ed7aaae4a7 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:10.028871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f0686c-ffe5-4bed-94f2-24dc608520c0 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:10.034898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a718d1-bde9-4e22-99ad-2851f4fd825b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:10.035524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a718d1-bde9-4e22-99ad-2851f4fd825b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:10.035813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a718d1-bde9-4e22-99ad-2851f4fd825b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:10.036339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a718d1-bde9-4e22-99ad-2851f4fd825b None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:10.036853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a718d1-bde9-4e22-99ad-2851f4fd825b None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:10.037167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a718d1-bde9-4e22-99ad-2851f4fd825b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:10.038603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a718d1-bde9-4e22-99ad-2851f4fd825b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:10.056202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 465/937] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:35:09 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:10.057063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a718d1-bde9-4e22-99ad-2851f4fd825b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:10.079715 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d02366d-74f0-4175-a355-8d07ad706d7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:10.080780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d02366d-74f0-4175-a355-8d07ad706d7a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:10.081365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d02366d-74f0-4175-a355-8d07ad706d7a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:10.081855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d02366d-74f0-4175-a355-8d07ad706d7a None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:10.083576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d02366d-74f0-4175-a355-8d07ad706d7a None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:10.084468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d02366d-74f0-4175-a355-8d07ad706d7a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:10.087071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d02366d-74f0-4175-a355-8d07ad706d7a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:10.101308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 473/938] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:09 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:10.103693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d02366d-74f0-4175-a355-8d07ad706d7a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:10.144875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:10.155695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:10.187734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a718d1-bde9-4e22-99ad-2851f4fd825b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:10.245449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7d0f6b-f99f-4240-8158-042790bffc9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:10.245449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7d0f6b-f99f-4240-8158-042790bffc9f None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:10.245748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7d0f6b-f99f-4240-8158-042790bffc9f None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:10.245837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7d0f6b-f99f-4240-8158-042790bffc9f None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:10.248104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7d0f6b-f99f-4240-8158-042790bffc9f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:10.248104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7d0f6b-f99f-4240-8158-042790bffc9f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:10.248304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7d0f6b-f99f-4240-8158-042790bffc9f None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:10.258851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 466/939] 10.222.0.40 () {56 vars in 1115 bytes} [Fri Jul 29 04:35:10 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:10.261427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d02366d-74f0-4175-a355-8d07ad706d7a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:10.266000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7d0f6b-f99f-4240-8158-042790bffc9f None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:10.303518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f2a8bd-a23c-4781-a1bc-c6f1a5cd8452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:10.304371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f2a8bd-a23c-4781-a1bc-c6f1a5cd8452 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:10.304694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f2a8bd-a23c-4781-a1bc-c6f1a5cd8452 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:10.304984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f2a8bd-a23c-4781-a1bc-c6f1a5cd8452 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:10.305758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f2a8bd-a23c-4781-a1bc-c6f1a5cd8452 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:10.307438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f2a8bd-a23c-4781-a1bc-c6f1a5cd8452 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:10.334345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 474/940] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:35:10 2022] POST /identity/v3/endpoints => generated 347 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:35:10.396867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7d0f6b-f99f-4240-8158-042790bffc9f None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:10.432010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:10.449285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ba5920-0c49-440d-9134-04533e3d1439 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:10.450373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba5920-0c49-440d-9134-04533e3d1439 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:10.450859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba5920-0c49-440d-9134-04533e3d1439 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:10.451383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba5920-0c49-440d-9134-04533e3d1439 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:10.464426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ba5920-0c49-440d-9134-04533e3d1439 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:10.466367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ba5920-0c49-440d-9134-04533e3d1439 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-b2ba5920-0c49-440d-9134-04533e3d1439 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:10.483327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:10.484387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:10.485347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:10.485347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:10.485347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:10.485347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:10.487685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 467/941] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:10.493275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f2a8bd-a23c-4781-a1bc-c6f1a5cd8452 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:10.587276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:10.601041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f2a8bd-a23c-4781-a1bc-c6f1a5cd8452 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:10.634514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2f2982-c922-48c3-a9c8-80c96558cfef None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:10.634847 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2f2982-c922-48c3-a9c8-80c96558cfef None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:10.635078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2f2982-c922-48c3-a9c8-80c96558cfef None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:10.636202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070ad37f-26d5-4642-895b-3555a62c1dc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:10.636854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070ad37f-26d5-4642-895b-3555a62c1dc9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:10.637139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070ad37f-26d5-4642-895b-3555a62c1dc9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:10.637419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070ad37f-26d5-4642-895b-3555a62c1dc9 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:10.638128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070ad37f-26d5-4642-895b-3555a62c1dc9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:10.638433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070ad37f-26d5-4642-895b-3555a62c1dc9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:10.639903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-070ad37f-26d5-4642-895b-3555a62c1dc9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:10.640406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 468/942] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:10.642162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb92d69-18e2-480b-af2f-ac55e9c404a9 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:10.642453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb92d69-18e2-480b-af2f-ac55e9c404a9 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:10.642732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb92d69-18e2-480b-af2f-ac55e9c404a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:10.656949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 475/943] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:10.662665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070ad37f-26d5-4642-895b-3555a62c1dc9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:10.664993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9fb92d69-18e2-480b-af2f-ac55e9c404a9 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:10.666203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9fb92d69-18e2-480b-af2f-ac55e9c404a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:10.748097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 469/944] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:10 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:10.767355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb92d69-18e2-480b-af2f-ac55e9c404a9 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:10.789741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070ad37f-26d5-4642-895b-3555a62c1dc9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:10.839351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf9ff4-d0b5-483d-aac0-ce1354435f9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:10.839921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf9ff4-d0b5-483d-aac0-ce1354435f9c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:10.840200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf9ff4-d0b5-483d-aac0-ce1354435f9c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:10.840474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cf9ff4-d0b5-483d-aac0-ce1354435f9c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:10.852625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cf9ff4-d0b5-483d-aac0-ce1354435f9c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:10.855255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cf9ff4-d0b5-483d-aac0-ce1354435f9c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-d6cf9ff4-d0b5-483d-aac0-ce1354435f9c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:35:10.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:10.872962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:10.873906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:10.873906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:10.873906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:35:10.873906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:35:10.873906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:35:10.873906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:35:10.873906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:35:10.873906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:35:10.873906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:35:10.873906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:10.876675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 476/945] 10.222.0.40 () {56 vars in 1117 bytes} [Fri Jul 29 04:35:10 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:10.880190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf9ff4-d0b5-483d-aac0-ce1354435f9c None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:10.896452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb92d69-18e2-480b-af2f-ac55e9c404a9 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:10.896842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:10.910516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:10.946024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7322b8-116a-4a4e-a66a-daccb7b8c4f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:10.947175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7322b8-116a-4a4e-a66a-daccb7b8c4f2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:10.947963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7322b8-116a-4a4e-a66a-daccb7b8c4f2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:10.948450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7322b8-116a-4a4e-a66a-daccb7b8c4f2 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:10.949318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7322b8-116a-4a4e-a66a-daccb7b8c4f2 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:10.951342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7322b8-116a-4a4e-a66a-daccb7b8c4f2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-4f7322b8-116a-4a4e-a66a-daccb7b8c4f2 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:10.963758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:10.964981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:10.965984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:10.969470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 470/946] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:10 2022] GET /identity/v3/services/object-store => generated 93 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:10.980631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7322b8-116a-4a4e-a66a-daccb7b8c4f2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:11.009141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cf9ff4-d0b5-483d-aac0-ce1354435f9c None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:11.046879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2357d20-e5b2-472d-9149-f6c08205d775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:11.047508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2357d20-e5b2-472d-9149-f6c08205d775 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:11.047791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2357d20-e5b2-472d-9149-f6c08205d775 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:11.048332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2357d20-e5b2-472d-9149-f6c08205d775 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:11.049169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2357d20-e5b2-472d-9149-f6c08205d775 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:11.049477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2357d20-e5b2-472d-9149-f6c08205d775 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:11.050906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2357d20-e5b2-472d-9149-f6c08205d775 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:11.071088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 477/947] 10.222.0.40 () {56 vars in 1149 bytes} [Fri Jul 29 04:35:10 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:11.076225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2357d20-e5b2-472d-9149-f6c08205d775 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:11.118572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7322b8-116a-4a4e-a66a-daccb7b8c4f2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:11.148865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:11.167483 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658b28ca-f08f-4d44-b4de-134339acaf16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:11.168836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658b28ca-f08f-4d44-b4de-134339acaf16 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:11.169416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658b28ca-f08f-4d44-b4de-134339acaf16 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:11.169972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-658b28ca-f08f-4d44-b4de-134339acaf16 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:11.171055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658b28ca-f08f-4d44-b4de-134339acaf16 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:11.171550 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658b28ca-f08f-4d44-b4de-134339acaf16 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:11.173796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-658b28ca-f08f-4d44-b4de-134339acaf16 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:11.185507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 471/948] 10.222.0.40 () {56 vars in 1122 bytes} [Fri Jul 29 04:35:10 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:11.188146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658b28ca-f08f-4d44-b4de-134339acaf16 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:11.192994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2357d20-e5b2-472d-9149-f6c08205d775 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:11.231727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46bcd56-bdb2-46b2-9bd3-591351087bc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:11.232367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bcd56-bdb2-46b2-9bd3-591351087bc6 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:11.232732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bcd56-bdb2-46b2-9bd3-591351087bc6 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:11.233014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46bcd56-bdb2-46b2-9bd3-591351087bc6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:11.237544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:11.246397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46bcd56-bdb2-46b2-9bd3-591351087bc6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:11.248175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b46bcd56-bdb2-46b2-9bd3-591351087bc6 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-b46bcd56-bdb2-46b2-9bd3-591351087bc6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:11.264211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:11.265205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:11.266069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:11.266069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:11.266069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:11.266069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:11.268176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 478/949] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:11.276659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46bcd56-bdb2-46b2-9bd3-591351087bc6 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:11.304381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-658b28ca-f08f-4d44-b4de-134339acaf16 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:11.348627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6cd18e-ca24-449b-ba05-bd64aca2d2c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:11.349780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6cd18e-ca24-449b-ba05-bd64aca2d2c5 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:11.350248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6cd18e-ca24-449b-ba05-bd64aca2d2c5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:11.350932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6cd18e-ca24-449b-ba05-bd64aca2d2c5 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:11.352207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6cd18e-ca24-449b-ba05-bd64aca2d2c5 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:11.352932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6cd18e-ca24-449b-ba05-bd64aca2d2c5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:11.355124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6cd18e-ca24-449b-ba05-bd64aca2d2c5 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:11.362913 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 472/950] 10.222.0.40 () {56 vars in 1122 bytes} [Fri Jul 29 04:35:11 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:11.369885 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6cd18e-ca24-449b-ba05-bd64aca2d2c5 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:11.399410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46bcd56-bdb2-46b2-9bd3-591351087bc6 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:11.418203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:11.438445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71d474e-ab40-47ad-9367-455317ded0ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:11.439074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71d474e-ab40-47ad-9367-455317ded0ea None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:11.439410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71d474e-ab40-47ad-9367-455317ded0ea None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:11.439697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71d474e-ab40-47ad-9367-455317ded0ea None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:11.440397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71d474e-ab40-47ad-9367-455317ded0ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:11.440749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71d474e-ab40-47ad-9367-455317ded0ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:11.442127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71d474e-ab40-47ad-9367-455317ded0ea None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:11.459558 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 479/951] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:11.460587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71d474e-ab40-47ad-9367-455317ded0ea None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:11.509577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6cd18e-ca24-449b-ba05-bd64aca2d2c5 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:11.509577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:11.543108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ff5b1c-7244-45c2-b773-1377623a2c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:11.543829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ff5b1c-7244-45c2-b773-1377623a2c88 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:11.544208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ff5b1c-7244-45c2-b773-1377623a2c88 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:11.544556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ff5b1c-7244-45c2-b773-1377623a2c88 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:11.545359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ff5b1c-7244-45c2-b773-1377623a2c88 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:11.547381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ff5b1c-7244-45c2-b773-1377623a2c88 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:11.573525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 473/952] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:35:11 2022] POST /identity/v3/endpoints => generated 349 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:35:11.591469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71d474e-ab40-47ad-9367-455317ded0ea None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:11.641543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09c9d6b-6b07-4f1a-bd68-b590316360a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:11.642116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09c9d6b-6b07-4f1a-bd68-b590316360a2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:11.642399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09c9d6b-6b07-4f1a-bd68-b590316360a2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:11.642727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09c9d6b-6b07-4f1a-bd68-b590316360a2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:11.652035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09c9d6b-6b07-4f1a-bd68-b590316360a2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:11.652379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09c9d6b-6b07-4f1a-bd68-b590316360a2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c09c9d6b-6b07-4f1a-bd68-b590316360a2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:35:11.665876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:11.666821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:11.667903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:35:11.667903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:35:11.667903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:35:11.667903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:35:11.667903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:35:11.667903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:35:11.667903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:11.667903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:11.667903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 480/953] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:11.675475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ff5b1c-7244-45c2-b773-1377623a2c88 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:11.793248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ff5b1c-7244-45c2-b773-1377623a2c88 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:11.831787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e1cbe5a-7596-4fb4-8ec0-d88f97e827d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:11.832449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1cbe5a-7596-4fb4-8ec0-d88f97e827d1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:11.832826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1cbe5a-7596-4fb4-8ec0-d88f97e827d1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:11.833174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e1cbe5a-7596-4fb4-8ec0-d88f97e827d1 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:11.833941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1cbe5a-7596-4fb4-8ec0-d88f97e827d1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:11.834253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1cbe5a-7596-4fb4-8ec0-d88f97e827d1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:11.835800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e1cbe5a-7596-4fb4-8ec0-d88f97e827d1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:11.848907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 474/954] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:35:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:11.854740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09c9d6b-6b07-4f1a-bd68-b590316360a2 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:11.895296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:11.970326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09c9d6b-6b07-4f1a-bd68-b590316360a2 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:12.006993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d424e50f-3cae-4315-8e5a-2a11c7416a80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:12.007751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d424e50f-3cae-4315-8e5a-2a11c7416a80 None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:12.008036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d424e50f-3cae-4315-8e5a-2a11c7416a80 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:12.008320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d424e50f-3cae-4315-8e5a-2a11c7416a80 None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/3f53019d35f244e4959d8340ba6f7e45/roles/8d6877d388f74e08b7653a068705e759` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:12.010615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d424e50f-3cae-4315-8e5a-2a11c7416a80 None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=3f53019d35f244e4959d8340ba6f7e45, role_id=8d6877d388f74e08b7653a068705e759)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:12.012193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d424e50f-3cae-4315-8e5a-2a11c7416a80 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:12.027778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 481/955] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:35:11 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/3f53019d35f244e4959d8340ba6f7e45/roles/8d6877d388f74e08b7653a068705e759 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:35:12.253262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a503ce42-6f82-424f-aa38-850c0903f521 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:12.254291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a503ce42-6f82-424f-aa38-850c0903f521 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:12.254784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a503ce42-6f82-424f-aa38-850c0903f521 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:12.256633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 475/956] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:12.262109 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e1293-133c-46f9-aaf5-a37ffb114bcb None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:12.262933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e1293-133c-46f9-aaf5-a37ffb114bcb None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:12.263413 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e1293-133c-46f9-aaf5-a37ffb114bcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:12.284014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-749e1293-133c-46f9-aaf5-a37ffb114bcb None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:12.285579 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-749e1293-133c-46f9-aaf5-a37ffb114bcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:12.357795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 482/957] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:12 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:12.381822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a503ce42-6f82-424f-aa38-850c0903f521 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:12.403668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:12.489737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d15a07d-b7b9-43f4-b9ff-247781f5c773 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:12.490077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d15a07d-b7b9-43f4-b9ff-247781f5c773 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:12.490288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d15a07d-b7b9-43f4-b9ff-247781f5c773 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:12.491986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 483/958] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:12.496885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d319e5-89b9-4849-93c7-c02173554ad9 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:12.497176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d319e5-89b9-4849-93c7-c02173554ad9 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:12.497460 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d319e5-89b9-4849-93c7-c02173554ad9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:12.500335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a503ce42-6f82-424f-aa38-850c0903f521 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:12.515993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-26d319e5-89b9-4849-93c7-c02173554ad9 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:12.517152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-26d319e5-89b9-4849-93c7-c02173554ad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:12.543621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:12.544568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:12.544898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:12.545334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:12.545697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:12.547070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:12.548756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:12.552108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WARNING keystone.server.flask.common [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 29 04:35:12.566829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.sql.core [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 29 04:35:12.566829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 29 04:35:12.566829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [parameters: {'user_id': 'd41da771ab5449bdafabace6e08723a2', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 29 04:35:12.566829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=92190) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application result = self._query(query) Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application conn.query(q) Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application result.read() Jul 29 04:35:12.576098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 29 04:35:12.577001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application next(self.gen) Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application yield resource Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application next(self.gen) Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self.session.rollback() Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self.force_reraise() Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise self.value Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application session.commit() Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 29 04:35:12.578511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._prepare_impl() Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self.session.flush() Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._flush(objects) Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application compat.raise_( Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application flush_context.execute() Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rec.execute(self) Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 29 04:35:12.579468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application result = connection._execute_20( Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 29 04:35:12.580379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application result = self._query(query) Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application conn.query(q) Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application result.read() Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 29 04:35:12.581414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [parameters: {'user_id': 'd41da771ab5449bdafabace6e08723a2', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:12.582437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Jul 29 04:35:12.583450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Jul 29 04:35:12.584549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Jul 29 04:35:12.584549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Jul 29 04:35:12.584549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 29 04:35:12.584549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 29 04:35:12.584549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 29 04:35:12.584549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:12.584549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 476/959] 10.222.0.40 () {60 vars in 1131 bytes} [Fri Jul 29 04:35:12 2022] POST /identity/v3/users => generated 160 bytes in 197 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:12.591110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:12.600734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 484/960] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:12 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:12.616761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d319e5-89b9-4849-93c7-c02173554ad9 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:12.708710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:12.733130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d319e5-89b9-4849-93c7-c02173554ad9 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:12.734346 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a04e1ae-7a2d-4a96-b901-6d5a69196aa5 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:12.771081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4305f2-bcbb-4f93-868f-50b91bf2b4d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:12.771456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1afbc-8962-4451-9883-3a2b691fb350 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:12.771648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4305f2-bcbb-4f93-868f-50b91bf2b4d1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:12.771938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4305f2-bcbb-4f93-868f-50b91bf2b4d1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:12.772240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4305f2-bcbb-4f93-868f-50b91bf2b4d1 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:12.773023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1afbc-8962-4451-9883-3a2b691fb350 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:12.773523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1afbc-8962-4451-9883-3a2b691fb350 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:12.774014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d1afbc-8962-4451-9883-3a2b691fb350 None admin] PATH_INFO: `/v3/services/compute` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:12.775004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d1afbc-8962-4451-9883-3a2b691fb350 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:12.777086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d1afbc-8962-4451-9883-3a2b691fb350 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:12.781369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4305f2-bcbb-4f93-868f-50b91bf2b4d1 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:12.783317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4305f2-bcbb-4f93-868f-50b91bf2b4d1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-26d1afbc-8962-4451-9883-3a2b691fb350 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:12.788733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:12.789764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 29 04:35:12.790641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-9e4305f2-bcbb-4f93-868f-50b91bf2b4d1 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:35:12.798316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:12.799477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:12.800655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:12.800655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:12.800655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:35:12.800655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:35:12.800655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:35:12.800655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:35:12.800655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:35:12.800655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:35:12.800655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 29 04:35:12.800655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:12.801458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 485/961] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:12 2022] GET /identity/v3/services/compute => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:12.802060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 477/962] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:35:12 2022] GET /identity/v3/users/cinder => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:12.803265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1afbc-8962-4451-9883-3a2b691fb350 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:12.808359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4305f2-bcbb-4f93-868f-50b91bf2b4d1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:12.829308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:12.936630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d1afbc-8962-4451-9883-3a2b691fb350 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:12.938661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4305f2-bcbb-4f93-868f-50b91bf2b4d1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:12.973711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5aa138-4bb9-4f62-9f3b-99d17ab27f3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:12.974396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5aa138-4bb9-4f62-9f3b-99d17ab27f3a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:12.974688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5aa138-4bb9-4f62-9f3b-99d17ab27f3a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:12.974967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5aa138-4bb9-4f62-9f3b-99d17ab27f3a None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:12.975736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5aa138-4bb9-4f62-9f3b-99d17ab27f3a None admin] RBAC: Adding query filter params (name=cinder) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:12.976047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5aa138-4bb9-4f62-9f3b-99d17ab27f3a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:12.977287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c5e3b0-be9a-440f-ba37-28e3ad26c562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:12.977504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5aa138-4bb9-4f62-9f3b-99d17ab27f3a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:12.978318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c5e3b0-be9a-440f-ba37-28e3ad26c562 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:12.978824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c5e3b0-be9a-440f-ba37-28e3ad26c562 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:12.979444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c5e3b0-be9a-440f-ba37-28e3ad26c562 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:12.980325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c5e3b0-be9a-440f-ba37-28e3ad26c562 None admin] RBAC: Adding query filter params (name=compute) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:12.980886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c5e3b0-be9a-440f-ba37-28e3ad26c562 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:12.982465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c5e3b0-be9a-440f-ba37-28e3ad26c562 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:12.988385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 486/963] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:35:12 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:12.994985 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c5e3b0-be9a-440f-ba37-28e3ad26c562 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:13.005599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:13.027796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 478/964] 10.222.0.40 () {56 vars in 1101 bytes} [Fri Jul 29 04:35:12 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:13.095637 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:13.116994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c5e3b0-be9a-440f-ba37-28e3ad26c562 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:13.155170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80beb8dd-fd94-45cc-9b80-bd674422853b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:13.156014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80beb8dd-fd94-45cc-9b80-bd674422853b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:13.156313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80beb8dd-fd94-45cc-9b80-bd674422853b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:13.156626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80beb8dd-fd94-45cc-9b80-bd674422853b None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:13.157379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80beb8dd-fd94-45cc-9b80-bd674422853b None admin] RBAC: Adding query filter params (type=compute) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:13.157694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80beb8dd-fd94-45cc-9b80-bd674422853b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:13.159127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80beb8dd-fd94-45cc-9b80-bd674422853b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:13.168982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 487/965] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:35:12 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:13.754242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b7fd07-e658-4d3c-8729-5c6a5c124507 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:13.754754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b7fd07-e658-4d3c-8729-5c6a5c124507 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:13.754830 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b7fd07-e658-4d3c-8729-5c6a5c124507 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:13.756303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 479/966] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:13.761328 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145f007c-9b2a-4f18-9d71-f228a795251e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:13.761804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145f007c-9b2a-4f18-9d71-f228a795251e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:13.761979 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145f007c-9b2a-4f18-9d71-f228a795251e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:13.781778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-145f007c-9b2a-4f18-9d71-f228a795251e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:13.782997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-145f007c-9b2a-4f18-9d71-f228a795251e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:13.854642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 488/967] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:13 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:13.876661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b7fd07-e658-4d3c-8729-5c6a5c124507 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:13.986180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b7fd07-e658-4d3c-8729-5c6a5c124507 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:14.019883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a23576-bdc7-4c2d-874c-42f28ffe3c30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:14.020529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a23576-bdc7-4c2d-874c-42f28ffe3c30 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.020848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a23576-bdc7-4c2d-874c-42f28ffe3c30 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.021132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a23576-bdc7-4c2d-874c-42f28ffe3c30 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.021761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a23576-bdc7-4c2d-874c-42f28ffe3c30 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:14.023161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a23576-bdc7-4c2d-874c-42f28ffe3c30 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-17a23576-bdc7-4c2d-874c-42f28ffe3c30 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:14.032817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:14.033841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:14.034706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:14.036083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 480/968] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:13 2022] GET /identity/v3/services/object-store => generated 93 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:14.039991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145f007c-9b2a-4f18-9d71-f228a795251e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:14.118245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:14.142800 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145f007c-9b2a-4f18-9d71-f228a795251e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:14.177657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b166d8e-8787-4978-86bb-84be4376b2ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:14.178640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b166d8e-8787-4978-86bb-84be4376b2ab None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.179149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b166d8e-8787-4978-86bb-84be4376b2ab None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.179630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b166d8e-8787-4978-86bb-84be4376b2ab None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.180574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b166d8e-8787-4978-86bb-84be4376b2ab None admin] RBAC: Adding query filter params (name=object-store) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:14.181140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b166d8e-8787-4978-86bb-84be4376b2ab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:14.182799 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b166d8e-8787-4978-86bb-84be4376b2ab None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:14.184070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9069812-585a-4147-8aa6-b633bb047480 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.184371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9069812-585a-4147-8aa6-b633bb047480 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.184679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9069812-585a-4147-8aa6-b633bb047480 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.186178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 481/969] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:14.191370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79302a30-3c87-4ab2-ae97-3c981b9637f4 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.191666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79302a30-3c87-4ab2-ae97-3c981b9637f4 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.191955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79302a30-3c87-4ab2-ae97-3c981b9637f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.192762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 489/970] 10.222.0.40 () {56 vars in 1122 bytes} [Fri Jul 29 04:35:14 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:14.194304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b166d8e-8787-4978-86bb-84be4376b2ab None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:14.210938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-79302a30-3c87-4ab2-ae97-3c981b9637f4 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:14.212478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-79302a30-3c87-4ab2-ae97-3c981b9637f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:14.275081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:14.289231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 482/971] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:14 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:14.304982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b166d8e-8787-4978-86bb-84be4376b2ab None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:14.313628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79302a30-3c87-4ab2-ae97-3c981b9637f4 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:14.344782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736eb7ab-806b-45dc-8f66-c4e940948eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:14.345641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736eb7ab-806b-45dc-8f66-c4e940948eb8 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.346080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736eb7ab-806b-45dc-8f66-c4e940948eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.346542 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736eb7ab-806b-45dc-8f66-c4e940948eb8 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.347371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736eb7ab-806b-45dc-8f66-c4e940948eb8 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:14.347828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736eb7ab-806b-45dc-8f66-c4e940948eb8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:14.349379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736eb7ab-806b-45dc-8f66-c4e940948eb8 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:14.359625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 490/972] 10.222.0.40 () {56 vars in 1122 bytes} [Fri Jul 29 04:35:14 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:14.360714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736eb7ab-806b-45dc-8f66-c4e940948eb8 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:14.364530 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:14.415350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79302a30-3c87-4ab2-ae97-3c981b9637f4 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:14.455595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bfae1a-dab9-4769-ac6b-c43dac7deb84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:14.456243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bfae1a-dab9-4769-ac6b-c43dac7deb84 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.456534 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bfae1a-dab9-4769-ac6b-c43dac7deb84 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.456851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bfae1a-dab9-4769-ac6b-c43dac7deb84 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.467196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bfae1a-dab9-4769-ac6b-c43dac7deb84 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:14.468642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736eb7ab-806b-45dc-8f66-c4e940948eb8 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-e2bfae1a-dab9-4769-ac6b-c43dac7deb84 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:14.473741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:14.474656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:14.481017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 483/973] 10.222.0.40 () {56 vars in 1120 bytes} [Fri Jul 29 04:35:14 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:14.481733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bfae1a-dab9-4769-ac6b-c43dac7deb84 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:14.504479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713ba075-d6ac-4dd3-a3cb-50829385236d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:14.505175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713ba075-d6ac-4dd3-a3cb-50829385236d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.505462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713ba075-d6ac-4dd3-a3cb-50829385236d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.505745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713ba075-d6ac-4dd3-a3cb-50829385236d None admin] PATH_INFO: `/v3/endpoints` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.506496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713ba075-d6ac-4dd3-a3cb-50829385236d None admin] RBAC: Adding query filter params (interface=admin, service_id=23fba158366349ea9b69cda5251a24fb, region_id=RegionOne) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:14.506809 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713ba075-d6ac-4dd3-a3cb-50829385236d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:14.508164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713ba075-d6ac-4dd3-a3cb-50829385236d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:14.513518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 491/974] 10.222.0.40 () {56 vars in 1249 bytes} [Fri Jul 29 04:35:14 2022] GET /identity/v3/endpoints?service_id=23fba158366349ea9b69cda5251a24fb&interface=admin®ion_id=RegionOne => generated 193 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:14.519545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713ba075-d6ac-4dd3-a3cb-50829385236d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:14.542491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:14.586108 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:14.597598 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bfae1a-dab9-4769-ac6b-c43dac7deb84 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:14.620470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713ba075-d6ac-4dd3-a3cb-50829385236d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:14.633160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f0f6f6-c779-42c7-9857-5752b4b2713b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:14.633790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f0f6f6-c779-42c7-9857-5752b4b2713b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.634073 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f0f6f6-c779-42c7-9857-5752b4b2713b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.634458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f0f6f6-c779-42c7-9857-5752b4b2713b None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.635294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f0f6f6-c779-42c7-9857-5752b4b2713b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:14.635601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f0f6f6-c779-42c7-9857-5752b4b2713b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:14.637152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f0f6f6-c779-42c7-9857-5752b4b2713b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:14.646542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 484/975] 10.222.0.40 () {56 vars in 1115 bytes} [Fri Jul 29 04:35:14 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:14.653777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f0f6f6-c779-42c7-9857-5752b4b2713b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:14.659643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b4cdbb-e1bb-4cfa-949a-f46115e8b55f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:14.660521 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b4cdbb-e1bb-4cfa-949a-f46115e8b55f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.661019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b4cdbb-e1bb-4cfa-949a-f46115e8b55f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.661477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b4cdbb-e1bb-4cfa-949a-f46115e8b55f None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.662307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b4cdbb-e1bb-4cfa-949a-f46115e8b55f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:14.663894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b4cdbb-e1bb-4cfa-949a-f46115e8b55f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:14.681079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 492/976] 10.222.0.40 () {56 vars in 1087 bytes} [Fri Jul 29 04:35:14 2022] GET /identity/v3/services => generated 1497 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:35:14.767692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f0f6f6-c779-42c7-9857-5752b4b2713b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:14.782981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:14.806490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318ee63a-29ce-49f3-af60-2edc981b5694 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:14.807070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318ee63a-29ce-49f3-af60-2edc981b5694 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.807361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318ee63a-29ce-49f3-af60-2edc981b5694 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.807641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318ee63a-29ce-49f3-af60-2edc981b5694 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.819898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318ee63a-29ce-49f3-af60-2edc981b5694 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:14.821403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318ee63a-29ce-49f3-af60-2edc981b5694 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-318ee63a-29ce-49f3-af60-2edc981b5694 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:14.837863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:14.838787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:14.839686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:14.839686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:14.839686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:14.839686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:14.839686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 485/977] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:14.845019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b4cdbb-e1bb-4cfa-949a-f46115e8b55f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:14.951358 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b4cdbb-e1bb-4cfa-949a-f46115e8b55f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:14.981334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:14.988321 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1954dd-8d3d-4378-b34a-395aab0a93ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:14.988984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1954dd-8d3d-4378-b34a-395aab0a93ba None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:14.989273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1954dd-8d3d-4378-b34a-395aab0a93ba None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:14.989557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1954dd-8d3d-4378-b34a-395aab0a93ba None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:14.990262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1954dd-8d3d-4378-b34a-395aab0a93ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:14.990572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1954dd-8d3d-4378-b34a-395aab0a93ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:14.991953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1954dd-8d3d-4378-b34a-395aab0a93ba None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:15.004098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 493/978] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:15.009475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318ee63a-29ce-49f3-af60-2edc981b5694 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:15.111827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318ee63a-29ce-49f3-af60-2edc981b5694 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:15.148056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d88f87-3e06-468b-9639-1e5087546cf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:15.148056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d88f87-3e06-468b-9639-1e5087546cf2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:15.148056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d88f87-3e06-468b-9639-1e5087546cf2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:15.148056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d88f87-3e06-468b-9639-1e5087546cf2 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:15.156861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d88f87-3e06-468b-9639-1e5087546cf2 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:15.158325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d88f87-3e06-468b-9639-1e5087546cf2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-05d88f87-3e06-468b-9639-1e5087546cf2 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:35:15.169673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:15.170599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:15.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:15.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:15.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:35:15.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:35:15.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:35:15.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:35:15.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:35:15.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:35:15.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 29 04:35:15.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:15.174388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 486/979] 10.222.0.40 () {56 vars in 1117 bytes} [Fri Jul 29 04:35:15 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:15.177539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1954dd-8d3d-4378-b34a-395aab0a93ba None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:15.209594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102aafe4-e2bd-468f-9ebf-1f0e951e37d0 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:15.209899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102aafe4-e2bd-468f-9ebf-1f0e951e37d0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:15.210148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102aafe4-e2bd-468f-9ebf-1f0e951e37d0 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:15.211510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 487/980] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:35:15.216299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8e9fd7-4e9f-49be-b72f-764c2b27d351 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:15.216590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8e9fd7-4e9f-49be-b72f-764c2b27d351 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:15.216900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8e9fd7-4e9f-49be-b72f-764c2b27d351 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:15.236592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-5c8e9fd7-4e9f-49be-b72f-764c2b27d351 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:15.237835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5c8e9fd7-4e9f-49be-b72f-764c2b27d351 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:15.259453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:15.290177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1954dd-8d3d-4378-b34a-395aab0a93ba None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:15.308222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 488/981] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:15 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:15.325794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb42f69-bee8-4e89-b6ce-8871130c72e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:15.326417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb42f69-bee8-4e89-b6ce-8871130c72e0 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:15.326747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb42f69-bee8-4e89-b6ce-8871130c72e0 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:15.326979 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb42f69-bee8-4e89-b6ce-8871130c72e0 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:15.328469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb42f69-bee8-4e89-b6ce-8871130c72e0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:15.328812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb42f69-bee8-4e89-b6ce-8871130c72e0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:15.330254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb42f69-bee8-4e89-b6ce-8871130c72e0 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:15.331254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8e9fd7-4e9f-49be-b72f-764c2b27d351 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:15.352055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 494/982] 10.222.0.40 () {56 vars in 1149 bytes} [Fri Jul 29 04:35:15 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:15.357523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb42f69-bee8-4e89-b6ce-8871130c72e0 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:15.410563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:15.439226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8e9fd7-4e9f-49be-b72f-764c2b27d351 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:15.463547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb42f69-bee8-4e89-b6ce-8871130c72e0 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:15.480426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b44d76-3a8b-42b2-ad16-531ae8a18878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:15.481186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b44d76-3a8b-42b2-ad16-531ae8a18878 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:15.481371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b44d76-3a8b-42b2-ad16-531ae8a18878 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:15.481714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b44d76-3a8b-42b2-ad16-531ae8a18878 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:15.482566 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b44d76-3a8b-42b2-ad16-531ae8a18878 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:15.484315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b44d76-3a8b-42b2-ad16-531ae8a18878 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:15.500020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 489/983] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:35:15 2022] POST /identity/v3/services => generated 242 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:35:15.508928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8331ebbb-739e-40e4-95d7-064f87ce8f14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:15.509575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8331ebbb-739e-40e4-95d7-064f87ce8f14 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:15.509868 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8331ebbb-739e-40e4-95d7-064f87ce8f14 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:15.510141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8331ebbb-739e-40e4-95d7-064f87ce8f14 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:15.587599 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8331ebbb-739e-40e4-95d7-064f87ce8f14 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:15.589033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8331ebbb-739e-40e4-95d7-064f87ce8f14 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:15.638485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:15.707869 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 495/984] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:15 2022] GET /identity/v3/auth/tokens => generated 1560 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 29 04:35:15.713693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b44d76-3a8b-42b2-ad16-531ae8a18878 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:15.733876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:15.822483 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b44d76-3a8b-42b2-ad16-531ae8a18878 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:15.856908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445033d7-1762-4f0f-9e18-02e8ca7cedaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:15.857558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445033d7-1762-4f0f-9e18-02e8ca7cedaa None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:15.857841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445033d7-1762-4f0f-9e18-02e8ca7cedaa None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:15.858116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445033d7-1762-4f0f-9e18-02e8ca7cedaa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:15.869319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445033d7-1762-4f0f-9e18-02e8ca7cedaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:15.870761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445033d7-1762-4f0f-9e18-02e8ca7cedaa None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-445033d7-1762-4f0f-9e18-02e8ca7cedaa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:15.885392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:15.886336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:15.887234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:15.887234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:15.887234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:15.887234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:15.888116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 490/985] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:15.892817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8331ebbb-739e-40e4-95d7-064f87ce8f14 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:15.999947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8331ebbb-739e-40e4-95d7-064f87ce8f14 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:16.034567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ea9cd3-6069-49cc-ab3f-ce72c7172554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:16.035286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ea9cd3-6069-49cc-ab3f-ce72c7172554 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:16.035584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ea9cd3-6069-49cc-ab3f-ce72c7172554 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:16.035869 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ea9cd3-6069-49cc-ab3f-ce72c7172554 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:16.036566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ea9cd3-6069-49cc-ab3f-ce72c7172554 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:16.036907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ea9cd3-6069-49cc-ab3f-ce72c7172554 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:16.038280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ea9cd3-6069-49cc-ab3f-ce72c7172554 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:16.054491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 496/986] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:16.055987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445033d7-1762-4f0f-9e18-02e8ca7cedaa None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:16.124927 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:16.160393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445033d7-1762-4f0f-9e18-02e8ca7cedaa None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:16.195078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b361c2-736d-434e-b4b1-3a486e02e592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:16.195725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b361c2-736d-434e-b4b1-3a486e02e592 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:16.196012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b361c2-736d-434e-b4b1-3a486e02e592 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:16.196292 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b361c2-736d-434e-b4b1-3a486e02e592 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:16.203191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b361c2-736d-434e-b4b1-3a486e02e592 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:16.204708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b361c2-736d-434e-b4b1-3a486e02e592 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-97b361c2-736d-434e-b4b1-3a486e02e592 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:35:16.216095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:16.217197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:16.218261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:35:16.218261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:35:16.218261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:35:16.218261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:35:16.218261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:35:16.218261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:35:16.218261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:16.218261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:16.220682 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 491/987] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:16.223174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ea9cd3-6069-49cc-ab3f-ce72c7172554 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:16.233282 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:16.331749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ea9cd3-6069-49cc-ab3f-ce72c7172554 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:16.365300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e369b0f3-3395-43ac-ac7f-35e835c421dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:16.365918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e369b0f3-3395-43ac-ac7f-35e835c421dc None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:16.366200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e369b0f3-3395-43ac-ac7f-35e835c421dc None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:16.366477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e369b0f3-3395-43ac-ac7f-35e835c421dc None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:16.367289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e369b0f3-3395-43ac-ac7f-35e835c421dc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:16.367602 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e369b0f3-3395-43ac-ac7f-35e835c421dc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:16.369253 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e369b0f3-3395-43ac-ac7f-35e835c421dc None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:16.382841 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 497/988] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:35:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:16.388615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b361c2-736d-434e-b4b1-3a486e02e592 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:16.494269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b361c2-736d-434e-b4b1-3a486e02e592 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:16.529006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:16.548309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563d933a-ac2a-44d4-9b56-3f4782c8dccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:16.548980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563d933a-ac2a-44d4-9b56-3f4782c8dccf None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:16.549267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563d933a-ac2a-44d4-9b56-3f4782c8dccf None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:16.549548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563d933a-ac2a-44d4-9b56-3f4782c8dccf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:16.550361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563d933a-ac2a-44d4-9b56-3f4782c8dccf None admin] RBAC: Adding query filter params (role.id=8d6877d388f74e08b7653a068705e759, scope.project.id=441bcfe715d44efea69d556677094857, user.id=3f53019d35f244e4959d8340ba6f7e45) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:16.550676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563d933a-ac2a-44d4-9b56-3f4782c8dccf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:16.552114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563d933a-ac2a-44d4-9b56-3f4782c8dccf None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:16.564689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 492/989] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:16 2022] GET /identity/v3/role_assignments?user.id=3f53019d35f244e4959d8340ba6f7e45&scope.project.id=441bcfe715d44efea69d556677094857&role.id=8d6877d388f74e08b7653a068705e759 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:16.735490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aea7968-05a4-41b1-8af1-05ebf62354bd None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:16.735848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aea7968-05a4-41b1-8af1-05ebf62354bd None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:16.736137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aea7968-05a4-41b1-8af1-05ebf62354bd None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:16.737608 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 498/990] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:16.742777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42286b05-b625-45dd-9f42-32b10982e082 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:16.743082 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42286b05-b625-45dd-9f42-32b10982e082 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:16.743371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42286b05-b625-45dd-9f42-32b10982e082 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:16.762029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-42286b05-b625-45dd-9f42-32b10982e082 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:16.763243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-42286b05-b625-45dd-9f42-32b10982e082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:16.833879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 493/991] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:16 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:16.856636 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aea7968-05a4-41b1-8af1-05ebf62354bd None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:16.924370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:16.964693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aea7968-05a4-41b1-8af1-05ebf62354bd None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:17.000651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217a20b9-4854-4f2e-bf22-5950c2b7ddaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:17.001313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217a20b9-4854-4f2e-bf22-5950c2b7ddaf None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:17.001600 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217a20b9-4854-4f2e-bf22-5950c2b7ddaf None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:17.001881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217a20b9-4854-4f2e-bf22-5950c2b7ddaf None admin] PATH_INFO: `/v3/services/object-store` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:17.002518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217a20b9-4854-4f2e-bf22-5950c2b7ddaf None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:17.003903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217a20b9-4854-4f2e-bf22-5950c2b7ddaf None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-217a20b9-4854-4f2e-bf22-5950c2b7ddaf None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:17.013075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:17.014056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:17.015047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 499/992] 10.222.0.40 () {56 vars in 1126 bytes} [Fri Jul 29 04:35:16 2022] GET /identity/v3/services/object-store => generated 93 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:17.022723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42286b05-b625-45dd-9f42-32b10982e082 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:17.126767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42286b05-b625-45dd-9f42-32b10982e082 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:17.150812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:17.164094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ab3d59-3d30-4305-9035-132d11bedab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:17.164763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ab3d59-3d30-4305-9035-132d11bedab0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:17.165054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ab3d59-3d30-4305-9035-132d11bedab0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:17.165334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ab3d59-3d30-4305-9035-132d11bedab0 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:17.166025 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ab3d59-3d30-4305-9035-132d11bedab0 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:17.166338 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ab3d59-3d30-4305-9035-132d11bedab0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:17.167710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ab3d59-3d30-4305-9035-132d11bedab0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:17.176830 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 494/993] 10.222.0.40 () {56 vars in 1122 bytes} [Fri Jul 29 04:35:17 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:17.179149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217a20b9-4854-4f2e-bf22-5950c2b7ddaf None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:17.288185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217a20b9-4854-4f2e-bf22-5950c2b7ddaf None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:17.317852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:17.325160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa69eb5c-e674-4d96-8310-6319ae959c78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:17.325788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa69eb5c-e674-4d96-8310-6319ae959c78 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:17.326072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa69eb5c-e674-4d96-8310-6319ae959c78 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:17.326353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa69eb5c-e674-4d96-8310-6319ae959c78 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:17.327039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa69eb5c-e674-4d96-8310-6319ae959c78 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:17.328543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa69eb5c-e674-4d96-8310-6319ae959c78 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:17.329950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa69eb5c-e674-4d96-8310-6319ae959c78 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:17.339709 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 500/994] 10.222.0.40 () {56 vars in 1122 bytes} [Fri Jul 29 04:35:17 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:17.340998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ab3d59-3d30-4305-9035-132d11bedab0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:17.446172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ab3d59-3d30-4305-9035-132d11bedab0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:17.480294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ee9583-bea7-41f6-bc86-40b0fc07c6d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:17.480947 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ee9583-bea7-41f6-bc86-40b0fc07c6d0 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:17.481236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ee9583-bea7-41f6-bc86-40b0fc07c6d0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:17.481519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ee9583-bea7-41f6-bc86-40b0fc07c6d0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:17.482228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ee9583-bea7-41f6-bc86-40b0fc07c6d0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:17.484180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0ee9583-bea7-41f6-bc86-40b0fc07c6d0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:17.501305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 495/995] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:35:17 2022] POST /identity/v3/endpoints => generated 325 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:35:17.569441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f84b2d4-8e7e-4f6e-84ba-4d74eced8d9e None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:17.569835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f84b2d4-8e7e-4f6e-84ba-4d74eced8d9e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:17.570195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f84b2d4-8e7e-4f6e-84ba-4d74eced8d9e None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:17.571692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 501/996] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:17.578208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65a3c33-40b4-4029-b87a-f739cc54e0d7 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:17.578503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65a3c33-40b4-4029-b87a-f739cc54e0d7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:17.578781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65a3c33-40b4-4029-b87a-f739cc54e0d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:17.598140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d65a3c33-40b4-4029-b87a-f739cc54e0d7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:17.599360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d65a3c33-40b4-4029-b87a-f739cc54e0d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:17.646988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:17.688049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 496/997] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:17 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:17.705775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f84b2d4-8e7e-4f6e-84ba-4d74eced8d9e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:17.813774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f84b2d4-8e7e-4f6e-84ba-4d74eced8d9e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:17.847504 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d1bf6-eb2b-4374-b458-994a4c0d6031 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:17.848144 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d1bf6-eb2b-4374-b458-994a4c0d6031 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:17.848425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d1bf6-eb2b-4374-b458-994a4c0d6031 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:17.848729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d1bf6-eb2b-4374-b458-994a4c0d6031 None admin] PATH_INFO: `/v3/services/compute` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:17.849409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3d1bf6-eb2b-4374-b458-994a4c0d6031 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:17.850834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3d1bf6-eb2b-4374-b458-994a4c0d6031 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-ff3d1bf6-eb2b-4374-b458-994a4c0d6031 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:17.860407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:17.861425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 29 04:35:17.862257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:17.864671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 502/998] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:17 2022] GET /identity/v3/services/compute => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:17.867814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65a3c33-40b4-4029-b87a-f739cc54e0d7 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:17.971428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65a3c33-40b4-4029-b87a-f739cc54e0d7 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:18.006541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fa414c-3ebb-41b8-8d6a-15ad5b114518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:18.007167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fa414c-3ebb-41b8-8d6a-15ad5b114518 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:18.007455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fa414c-3ebb-41b8-8d6a-15ad5b114518 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:18.007734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fa414c-3ebb-41b8-8d6a-15ad5b114518 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:18.008419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fa414c-3ebb-41b8-8d6a-15ad5b114518 None admin] RBAC: Adding query filter params (name=compute) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:18.008769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fa414c-3ebb-41b8-8d6a-15ad5b114518 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:18.010148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fa414c-3ebb-41b8-8d6a-15ad5b114518 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:18.019833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 497/999] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:35:17 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:18.021409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d1bf6-eb2b-4374-b458-994a4c0d6031 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:18.084350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:18.127139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d1bf6-eb2b-4374-b458-994a4c0d6031 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:18.162787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7b9e29-6310-430e-8c02-6a98576529c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:18.163414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7b9e29-6310-430e-8c02-6a98576529c4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:18.163696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7b9e29-6310-430e-8c02-6a98576529c4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:18.163970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7b9e29-6310-430e-8c02-6a98576529c4 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:18.164662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7b9e29-6310-430e-8c02-6a98576529c4 None admin] RBAC: Adding query filter params (type=compute) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:18.164971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7b9e29-6310-430e-8c02-6a98576529c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:18.166333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7b9e29-6310-430e-8c02-6a98576529c4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:18.176665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 503/1000] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:35:18 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:18.178244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fa414c-3ebb-41b8-8d6a-15ad5b114518 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:18.215385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:18.285417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fa414c-3ebb-41b8-8d6a-15ad5b114518 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:18.320330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d65e154-802e-49fc-8fe5-e957171a36eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:18.320991 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d65e154-802e-49fc-8fe5-e957171a36eb None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:18.321278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d65e154-802e-49fc-8fe5-e957171a36eb None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:18.321556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d65e154-802e-49fc-8fe5-e957171a36eb None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:18.322309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d65e154-802e-49fc-8fe5-e957171a36eb None admin] RBAC: Adding query filter params (interface=public, service_id=c1e761d935334818ac3550fe15aa4b20, region_id=RegionOne) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:18.322621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d65e154-802e-49fc-8fe5-e957171a36eb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:18.323989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d65e154-802e-49fc-8fe5-e957171a36eb None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:18.333769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 498/1001] 10.222.0.40 () {56 vars in 1251 bytes} [Fri Jul 29 04:35:18 2022] GET /identity/v3/endpoints?service_id=c1e761d935334818ac3550fe15aa4b20&interface=public®ion_id=RegionOne => generated 194 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:18.335110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7b9e29-6310-430e-8c02-6a98576529c4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:18.438073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7b9e29-6310-430e-8c02-6a98576529c4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:18.460941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:18.474418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6318c51-f594-47e2-aef8-c9bdaf824a4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:18.475046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6318c51-f594-47e2-aef8-c9bdaf824a4c None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:18.475336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6318c51-f594-47e2-aef8-c9bdaf824a4c None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:18.475620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6318c51-f594-47e2-aef8-c9bdaf824a4c None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:18.476279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6318c51-f594-47e2-aef8-c9bdaf824a4c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:18.477686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6318c51-f594-47e2-aef8-c9bdaf824a4c None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:18.492628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 504/1002] 10.222.0.40 () {56 vars in 1087 bytes} [Fri Jul 29 04:35:18 2022] GET /identity/v3/services => generated 1727 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:35:18.639322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f70c4a-977e-4f0c-9633-7ca5bcbb6de7 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:18.639536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f70c4a-977e-4f0c-9633-7ca5bcbb6de7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:18.640421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f70c4a-977e-4f0c-9633-7ca5bcbb6de7 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:18.642099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 499/1003] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:18.647138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31734310-f40f-4ae0-a078-79df7c722dff None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:18.647440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31734310-f40f-4ae0-a078-79df7c722dff None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:18.650584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31734310-f40f-4ae0-a078-79df7c722dff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:18.670714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-31734310-f40f-4ae0-a078-79df7c722dff None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:18.672033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-31734310-f40f-4ae0-a078-79df7c722dff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:18.743235 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 505/1004] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:18 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:18.765843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f70c4a-977e-4f0c-9633-7ca5bcbb6de7 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:18.866315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:18.871679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f70c4a-977e-4f0c-9633-7ca5bcbb6de7 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:18.908371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35033a47-5b18-43fe-a5ef-8ab789ca67f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:18.908961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35033a47-5b18-43fe-a5ef-8ab789ca67f0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:18.909251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35033a47-5b18-43fe-a5ef-8ab789ca67f0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:18.909532 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35033a47-5b18-43fe-a5ef-8ab789ca67f0 None admin] PATH_INFO: `/v3/services/image` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:18.910163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35033a47-5b18-43fe-a5ef-8ab789ca67f0 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:18.911537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35033a47-5b18-43fe-a5ef-8ab789ca67f0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-35033a47-5b18-43fe-a5ef-8ab789ca67f0 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:18.920806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:18.922132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:18.928143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 500/1005] 10.222.0.40 () {56 vars in 1105 bytes} [Fri Jul 29 04:35:18 2022] GET /identity/v3/services/image => generated 86 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:18.930665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31734310-f40f-4ae0-a078-79df7c722dff None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:19.041649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31734310-f40f-4ae0-a078-79df7c722dff None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:19.058443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:19.077278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe7280a-a16a-4098-bdf3-212eb1ae3b2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:19.078038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe7280a-a16a-4098-bdf3-212eb1ae3b2f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:19.078320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe7280a-a16a-4098-bdf3-212eb1ae3b2f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:19.078598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe7280a-a16a-4098-bdf3-212eb1ae3b2f None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:19.079288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe7280a-a16a-4098-bdf3-212eb1ae3b2f None admin] RBAC: Adding query filter params (name=image) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:19.079591 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe7280a-a16a-4098-bdf3-212eb1ae3b2f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:19.081030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe7280a-a16a-4098-bdf3-212eb1ae3b2f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:19.090322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 506/1006] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:18 2022] GET /identity/v3/services?name=image => generated 122 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:19.091444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35033a47-5b18-43fe-a5ef-8ab789ca67f0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:19.195083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35033a47-5b18-43fe-a5ef-8ab789ca67f0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:19.228775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b926afb6-aa3a-4353-a1a0-12ba52808717 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:19.229401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b926afb6-aa3a-4353-a1a0-12ba52808717 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:19.229687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b926afb6-aa3a-4353-a1a0-12ba52808717 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:19.230027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b926afb6-aa3a-4353-a1a0-12ba52808717 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:19.230715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b926afb6-aa3a-4353-a1a0-12ba52808717 None admin] RBAC: Adding query filter params (type=image) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:19.231022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b926afb6-aa3a-4353-a1a0-12ba52808717 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:19.232392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b926afb6-aa3a-4353-a1a0-12ba52808717 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:19.244628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 501/1007] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:19 2022] GET /identity/v3/services?type=image => generated 122 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:19.533016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e727d5-aaaa-45db-b478-16f3dfb55e7a None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:19.533391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e727d5-aaaa-45db-b478-16f3dfb55e7a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:19.533573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e727d5-aaaa-45db-b478-16f3dfb55e7a None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:19.535056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 507/1008] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:19.539966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f218f5-a48b-4b16-a699-8db909bbe591 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:19.540263 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f218f5-a48b-4b16-a699-8db909bbe591 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:19.540548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f218f5-a48b-4b16-a699-8db909bbe591 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:19.558817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a7f218f5-a48b-4b16-a699-8db909bbe591 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:19.560010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a7f218f5-a48b-4b16-a699-8db909bbe591 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:19.584520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:19.631534 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 502/1009] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:19 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:19.654348 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e727d5-aaaa-45db-b478-16f3dfb55e7a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:19.761107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e727d5-aaaa-45db-b478-16f3dfb55e7a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:19.795532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d363dc0-f62b-407c-8937-db17a2ff9dc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:19.796132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d363dc0-f62b-407c-8937-db17a2ff9dc1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:19.796422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d363dc0-f62b-407c-8937-db17a2ff9dc1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:19.796731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d363dc0-f62b-407c-8937-db17a2ff9dc1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:19.798144 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d363dc0-f62b-407c-8937-db17a2ff9dc1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:19.799575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d363dc0-f62b-407c-8937-db17a2ff9dc1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:19.806149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 508/1010] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:19.807092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f218f5-a48b-4b16-a699-8db909bbe591 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:19.905862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f218f5-a48b-4b16-a699-8db909bbe591 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:19.940823 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa3ae02-5433-41ec-8c0b-330ef2b9ece3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:19.941453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa3ae02-5433-41ec-8c0b-330ef2b9ece3 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:19.941744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa3ae02-5433-41ec-8c0b-330ef2b9ece3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:19.942030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa3ae02-5433-41ec-8c0b-330ef2b9ece3 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:19.942731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa3ae02-5433-41ec-8c0b-330ef2b9ece3 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:19.944166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa3ae02-5433-41ec-8c0b-330ef2b9ece3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:19.977630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 503/1011] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:35:19 2022] POST /identity/v3/projects => generated 311 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:35:20.515711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4086713-b6eb-4786-b368-69e310b78c0c None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:20.516229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4086713-b6eb-4786-b368-69e310b78c0c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:20.516307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4086713-b6eb-4786-b368-69e310b78c0c None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:20.517681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 509/1012] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:20.522714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2155ab2f-741b-423f-abb7-2f4b2b7146e6 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:20.523020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2155ab2f-741b-423f-abb7-2f4b2b7146e6 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:20.523307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2155ab2f-741b-423f-abb7-2f4b2b7146e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:20.541798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2155ab2f-741b-423f-abb7-2f4b2b7146e6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:20.543002 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2155ab2f-741b-423f-abb7-2f4b2b7146e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:20.546361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:20.616815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 504/1013] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:20 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:20.635278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4086713-b6eb-4786-b368-69e310b78c0c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:20.695367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:20.749246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4086713-b6eb-4786-b368-69e310b78c0c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:20.784880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2401df86-a7d4-4c03-8e33-fc8f6061f322 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:20.785455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2401df86-a7d4-4c03-8e33-fc8f6061f322 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:20.785742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2401df86-a7d4-4c03-8e33-fc8f6061f322 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:20.786021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2401df86-a7d4-4c03-8e33-fc8f6061f322 None admin] PATH_INFO: `/v3/services/compute` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:20.786674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2401df86-a7d4-4c03-8e33-fc8f6061f322 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:20.788051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2401df86-a7d4-4c03-8e33-fc8f6061f322 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-2401df86-a7d4-4c03-8e33-fc8f6061f322 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:20.798495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:20.799515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 29 04:35:20.800385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:20.801801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 510/1014] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:20 2022] GET /identity/v3/services/compute => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:20.805681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2155ab2f-741b-423f-abb7-2f4b2b7146e6 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:20.905715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2155ab2f-741b-423f-abb7-2f4b2b7146e6 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:20.940800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4151bd5-4b4a-4548-9dd4-7d479dfdaa0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:20.942189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4151bd5-4b4a-4548-9dd4-7d479dfdaa0c None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:20.942681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4151bd5-4b4a-4548-9dd4-7d479dfdaa0c None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:20.943168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4151bd5-4b4a-4548-9dd4-7d479dfdaa0c None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:20.944092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4151bd5-4b4a-4548-9dd4-7d479dfdaa0c None admin] RBAC: Adding query filter params (name=compute) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:20.944587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4151bd5-4b4a-4548-9dd4-7d479dfdaa0c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:20.946178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4151bd5-4b4a-4548-9dd4-7d479dfdaa0c None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:20.955901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 505/1015] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:35:20 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:20.957226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2401df86-a7d4-4c03-8e33-fc8f6061f322 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:21.067933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2401df86-a7d4-4c03-8e33-fc8f6061f322 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:21.081063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:21.106029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5599443d-60a4-42a3-b3f9-a905db872bef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:21.106666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5599443d-60a4-42a3-b3f9-a905db872bef None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:21.106953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5599443d-60a4-42a3-b3f9-a905db872bef None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:21.107234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5599443d-60a4-42a3-b3f9-a905db872bef None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:21.107927 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5599443d-60a4-42a3-b3f9-a905db872bef None admin] RBAC: Adding query filter params (type=compute) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:21.108364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5599443d-60a4-42a3-b3f9-a905db872bef None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:21.110323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5599443d-60a4-42a3-b3f9-a905db872bef None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:21.120325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 511/1016] 10.222.0.40 () {56 vars in 1112 bytes} [Fri Jul 29 04:35:20 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:21.121466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4151bd5-4b4a-4548-9dd4-7d479dfdaa0c None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:21.133015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:21.229353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4151bd5-4b4a-4548-9dd4-7d479dfdaa0c None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:21.252786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1200e124-c51e-4b23-b175-e1bc9bf1f712 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:21.253072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1200e124-c51e-4b23-b175-e1bc9bf1f712 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:21.253349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1200e124-c51e-4b23-b175-e1bc9bf1f712 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:21.254685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 512/1017] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:35:21.259506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2afc02b-d1fd-41b9-8117-d17cd25f52c6 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:21.259790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2afc02b-d1fd-41b9-8117-d17cd25f52c6 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:21.260229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2afc02b-d1fd-41b9-8117-d17cd25f52c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:21.267817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c182bdba-7082-4663-815e-c70e1eb37ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:21.268442 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c182bdba-7082-4663-815e-c70e1eb37ce7 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:21.268772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c182bdba-7082-4663-815e-c70e1eb37ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:21.269050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c182bdba-7082-4663-815e-c70e1eb37ce7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:21.269770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c182bdba-7082-4663-815e-c70e1eb37ce7 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:21.271127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c182bdba-7082-4663-815e-c70e1eb37ce7 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:21.278711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-a2afc02b-d1fd-41b9-8117-d17cd25f52c6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:21.279901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a2afc02b-d1fd-41b9-8117-d17cd25f52c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:21.298200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 506/1018] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:35:21 2022] POST /identity/v3/endpoints => generated 334 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:35:21.368105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 513/1019] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:21 2022] POST /identity/v3/auth/tokens => generated 1863 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:21.393252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c182bdba-7082-4663-815e-c70e1eb37ce7 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:21.462661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:21.508508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c182bdba-7082-4663-815e-c70e1eb37ce7 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:21.546449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39086e7f-3748-419f-8b25-c07215e2fb96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:21.547220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39086e7f-3748-419f-8b25-c07215e2fb96 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:21.547513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39086e7f-3748-419f-8b25-c07215e2fb96 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:21.548051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39086e7f-3748-419f-8b25-c07215e2fb96 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:21.548809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39086e7f-3748-419f-8b25-c07215e2fb96 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:21.550194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39086e7f-3748-419f-8b25-c07215e2fb96 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:21.569350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 507/1020] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:35:21 2022] POST /identity/v3/services => generated 242 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:35:22.003931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf95f56-46d8-4bde-a57d-5e5d39bee83f None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:22.004451 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf95f56-46d8-4bde-a57d-5e5d39bee83f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:22.004554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf95f56-46d8-4bde-a57d-5e5d39bee83f None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:22.005894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 514/1021] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:22.010937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d163ab-f875-47a3-a5dd-83f31e30efc2 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:22.011338 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d163ab-f875-47a3-a5dd-83f31e30efc2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:22.011498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d163ab-f875-47a3-a5dd-83f31e30efc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:22.029543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-15d163ab-f875-47a3-a5dd-83f31e30efc2 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:22.032210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-15d163ab-f875-47a3-a5dd-83f31e30efc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:22.107573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 508/1022] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:22 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:22.131820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf95f56-46d8-4bde-a57d-5e5d39bee83f None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:22.239353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf95f56-46d8-4bde-a57d-5e5d39bee83f None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:22.241146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:22.276505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4ff5d7-9e72-4964-ab02-ed2651e93ff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:22.277157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ff5d7-9e72-4964-ab02-ed2651e93ff3 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:22.277449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ff5d7-9e72-4964-ab02-ed2651e93ff3 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:22.277729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4ff5d7-9e72-4964-ab02-ed2651e93ff3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:22.278891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4ff5d7-9e72-4964-ab02-ed2651e93ff3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:22.281218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4ff5d7-9e72-4964-ab02-ed2651e93ff3 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:22.287289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 515/1023] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:22.288548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d163ab-f875-47a3-a5dd-83f31e30efc2 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:22.336938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:22.393207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d163ab-f875-47a3-a5dd-83f31e30efc2 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:22.427030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40201bc8-6622-4833-8431-73749f73d5a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:22.427651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40201bc8-6622-4833-8431-73749f73d5a8 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:22.427943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40201bc8-6622-4833-8431-73749f73d5a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:22.428224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40201bc8-6622-4833-8431-73749f73d5a8 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:22.428951 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40201bc8-6622-4833-8431-73749f73d5a8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:22.430378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40201bc8-6622-4833-8431-73749f73d5a8 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:22.453820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 509/1024] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:35:22 2022] POST /identity/v3/users => generated 285 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:35:23.427503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3375d90b-28e6-49f2-98d1-e80d88c8ea79 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:23.428063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3375d90b-28e6-49f2-98d1-e80d88c8ea79 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:23.428063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3375d90b-28e6-49f2-98d1-e80d88c8ea79 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:23.429495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 516/1025] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:23.434545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9887cc17-ba9b-4bbd-8566-304cf1c137b3 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:23.435015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9887cc17-ba9b-4bbd-8566-304cf1c137b3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:23.435176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9887cc17-ba9b-4bbd-8566-304cf1c137b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:23.453445 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9887cc17-ba9b-4bbd-8566-304cf1c137b3 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:23.454653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9887cc17-ba9b-4bbd-8566-304cf1c137b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:23.523006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 510/1026] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:23 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:23.546185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3375d90b-28e6-49f2-98d1-e80d88c8ea79 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:23.646731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b773c22-fa8d-49fc-b9b4-4d4eaa0a5100 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:23.647052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b773c22-fa8d-49fc-b9b4-4d4eaa0a5100 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:23.647312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b773c22-fa8d-49fc-b9b4-4d4eaa0a5100 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:23.648706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 511/1027] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:35:23.658048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06829366-01f8-4d20-8dae-688bbd32eaff None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:23.658183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3375d90b-28e6-49f2-98d1-e80d88c8ea79 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:23.658645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06829366-01f8-4d20-8dae-688bbd32eaff None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:23.659100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06829366-01f8-4d20-8dae-688bbd32eaff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:23.677381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-06829366-01f8-4d20-8dae-688bbd32eaff None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:23.678930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-06829366-01f8-4d20-8dae-688bbd32eaff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:23.699994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a6ce6-2c13-4be5-b5d6-4447cee21a4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:23.700674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a6ce6-2c13-4be5-b5d6-4447cee21a4b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:23.700964 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a6ce6-2c13-4be5-b5d6-4447cee21a4b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:23.701245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a6ce6-2c13-4be5-b5d6-4447cee21a4b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:23.705436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:23.713656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879a6ce6-2c13-4be5-b5d6-4447cee21a4b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-879a6ce6-2c13-4be5-b5d6-4447cee21a4b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:23.718645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:23.719798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:23.720663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 517/1028] 10.222.0.40 () {56 vars in 1120 bytes} [Fri Jul 29 04:35:23 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:23.726477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a6ce6-2c13-4be5-b5d6-4447cee21a4b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:23.754516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 512/1029] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:23 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:23.761655 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:23.777110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06829366-01f8-4d20-8dae-688bbd32eaff None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:23.833211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a6ce6-2c13-4be5-b5d6-4447cee21a4b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:23.868657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925224f7-6301-4a0b-b22a-2dabd713e517 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:23.869283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925224f7-6301-4a0b-b22a-2dabd713e517 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:23.869571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925224f7-6301-4a0b-b22a-2dabd713e517 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:23.869852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925224f7-6301-4a0b-b22a-2dabd713e517 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:23.870572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925224f7-6301-4a0b-b22a-2dabd713e517 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:23.870881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925224f7-6301-4a0b-b22a-2dabd713e517 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:23.872255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925224f7-6301-4a0b-b22a-2dabd713e517 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:23.878331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06829366-01f8-4d20-8dae-688bbd32eaff None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:23.881205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 518/1030] 10.222.0.40 () {56 vars in 1115 bytes} [Fri Jul 29 04:35:23 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:23.886674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925224f7-6301-4a0b-b22a-2dabd713e517 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:23.913706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0172dca-c43d-4ee7-8688-ec4a3893b286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:23.914378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0172dca-c43d-4ee7-8688-ec4a3893b286 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:23.914658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0172dca-c43d-4ee7-8688-ec4a3893b286 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:23.914907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0172dca-c43d-4ee7-8688-ec4a3893b286 None admin] PATH_INFO: `/v3/services/image` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:23.915606 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0172dca-c43d-4ee7-8688-ec4a3893b286 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:23.917368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0172dca-c43d-4ee7-8688-ec4a3893b286 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-c0172dca-c43d-4ee7-8688-ec4a3893b286 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:23.927585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:23.928619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:23.934180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 513/1031] 10.222.0.40 () {56 vars in 1105 bytes} [Fri Jul 29 04:35:23 2022] GET /identity/v3/services/image => generated 86 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:23.935180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0172dca-c43d-4ee7-8688-ec4a3893b286 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:23.963625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:23.993170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:24.004804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925224f7-6301-4a0b-b22a-2dabd713e517 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:24.041380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c1ea9e-5ee2-46fb-bea0-8fdf920452c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:24.042034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c1ea9e-5ee2-46fb-bea0-8fdf920452c0 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.042322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c1ea9e-5ee2-46fb-bea0-8fdf920452c0 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.042603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c1ea9e-5ee2-46fb-bea0-8fdf920452c0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.046076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0172dca-c43d-4ee7-8688-ec4a3893b286 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:24.054688 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c1ea9e-5ee2-46fb-bea0-8fdf920452c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:24.056151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c1ea9e-5ee2-46fb-bea0-8fdf920452c0 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-a0c1ea9e-5ee2-46fb-bea0-8fdf920452c0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:24.071385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:24.072456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:24.073407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:24.073407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:24.073407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:24.073407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:24.073407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 519/1032] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:24.079997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c1ea9e-5ee2-46fb-bea0-8fdf920452c0 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:24.090079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4a4588-9ab5-4ec8-8369-d7fefc0589f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:24.090711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4a4588-9ab5-4ec8-8369-d7fefc0589f2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.090996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4a4588-9ab5-4ec8-8369-d7fefc0589f2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.091272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4a4588-9ab5-4ec8-8369-d7fefc0589f2 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.091950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4a4588-9ab5-4ec8-8369-d7fefc0589f2 None admin] RBAC: Adding query filter params (name=image) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:24.092256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4a4588-9ab5-4ec8-8369-d7fefc0589f2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:24.093710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4a4588-9ab5-4ec8-8369-d7fefc0589f2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:24.103287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 514/1033] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:23 2022] GET /identity/v3/services?name=image => generated 122 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:24.104272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4a4588-9ab5-4ec8-8369-d7fefc0589f2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:24.196024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c1ea9e-5ee2-46fb-bea0-8fdf920452c0 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:24.208276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:24.220550 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4a4588-9ab5-4ec8-8369-d7fefc0589f2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:24.234400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e258305f-2df0-45d8-9fec-1043cc05feeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:24.235131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e258305f-2df0-45d8-9fec-1043cc05feeb None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.235900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e258305f-2df0-45d8-9fec-1043cc05feeb None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.235900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e258305f-2df0-45d8-9fec-1043cc05feeb None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.236471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e258305f-2df0-45d8-9fec-1043cc05feeb None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:24.236798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e258305f-2df0-45d8-9fec-1043cc05feeb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:24.238393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e258305f-2df0-45d8-9fec-1043cc05feeb None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:24.251899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 520/1034] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:24.257374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e258305f-2df0-45d8-9fec-1043cc05feeb None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:24.261660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:24.263873 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3568d560-e8d4-4c14-990b-eba2b61d4919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:24.264632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3568d560-e8d4-4c14-990b-eba2b61d4919 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.264848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3568d560-e8d4-4c14-990b-eba2b61d4919 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.265175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3568d560-e8d4-4c14-990b-eba2b61d4919 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.265824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3568d560-e8d4-4c14-990b-eba2b61d4919 None admin] RBAC: Adding query filter params (type=image) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:24.266139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3568d560-e8d4-4c14-990b-eba2b61d4919 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:24.267546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3568d560-e8d4-4c14-990b-eba2b61d4919 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:24.273005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 515/1035] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:24 2022] GET /identity/v3/services?type=image => generated 350 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:24.278657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3568d560-e8d4-4c14-990b-eba2b61d4919 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:24.370335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e258305f-2df0-45d8-9fec-1043cc05feeb None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:24.382978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3568d560-e8d4-4c14-990b-eba2b61d4919 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:24.408431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb8167f-3411-4fba-8975-bb87a8f130ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:24.409439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb8167f-3411-4fba-8975-bb87a8f130ea None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.409973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb8167f-3411-4fba-8975-bb87a8f130ea None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.410532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb8167f-3411-4fba-8975-bb87a8f130ea None admin] PATH_INFO: `/v3/users/nova` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.417841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03995e-bd40-446c-88e2-50a9310bca70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:24.418248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb8167f-3411-4fba-8975-bb87a8f130ea None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:24.418491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03995e-bd40-446c-88e2-50a9310bca70 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.418778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03995e-bd40-446c-88e2-50a9310bca70 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.419062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03995e-bd40-446c-88e2-50a9310bca70 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.419764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb8167f-3411-4fba-8975-bb87a8f130ea None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:24.419864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e03995e-bd40-446c-88e2-50a9310bca70 None admin] RBAC: Adding query filter params (interface=public, service_id=8a13c813fdf24c0484f205b3309dec68, region_id=RegionOne) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:24.420154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e03995e-bd40-446c-88e2-50a9310bca70 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:24.421779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e03995e-bd40-446c-88e2-50a9310bca70 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:24.427497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 516/1036] 10.222.0.40 () {56 vars in 1251 bytes} [Fri Jul 29 04:35:24 2022] GET /identity/v3/endpoints?service_id=8a13c813fdf24c0484f205b3309dec68&interface=public®ion_id=RegionOne => generated 194 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:24.432964 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03995e-bd40-446c-88e2-50a9310bca70 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-0fb8167f-3411-4fba-8975-bb87a8f130ea None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:35:24.436271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:24.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:24.438541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 521/1037] 10.222.0.40 () {56 vars in 1093 bytes} [Fri Jul 29 04:35:24 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:24.442955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb8167f-3411-4fba-8975-bb87a8f130ea None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:24.457194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:24.508735 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:24.544326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03995e-bd40-446c-88e2-50a9310bca70 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:24.551309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb8167f-3411-4fba-8975-bb87a8f130ea None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:24.578374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffabe69e-4fbb-4cf3-93d7-c7e7aa77e142 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:24.578991 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabe69e-4fbb-4cf3-93d7-c7e7aa77e142 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.579280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabe69e-4fbb-4cf3-93d7-c7e7aa77e142 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.579561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffabe69e-4fbb-4cf3-93d7-c7e7aa77e142 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.580217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffabe69e-4fbb-4cf3-93d7-c7e7aa77e142 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:24.581626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffabe69e-4fbb-4cf3-93d7-c7e7aa77e142 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:24.585093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc6c088-b63f-4073-a633-1d99e9a02047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:24.585708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc6c088-b63f-4073-a633-1d99e9a02047 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.585988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc6c088-b63f-4073-a633-1d99e9a02047 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.586263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc6c088-b63f-4073-a633-1d99e9a02047 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.587049 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc6c088-b63f-4073-a633-1d99e9a02047 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:24.587355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc6c088-b63f-4073-a633-1d99e9a02047 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:24.588380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 517/1038] 10.222.0.40 () {56 vars in 1087 bytes} [Fri Jul 29 04:35:24 2022] GET /identity/v3/services => generated 1957 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:35:24.588819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc6c088-b63f-4073-a633-1d99e9a02047 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:24.589436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d787e2-d702-4b53-96e4-031df6de0231 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.589732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d787e2-d702-4b53-96e4-031df6de0231 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.590016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d787e2-d702-4b53-96e4-031df6de0231 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.591260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 518/1039] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:24.602544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490758ad-0b22-45d7-85fb-5550439c5e42 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.602865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490758ad-0b22-45d7-85fb-5550439c5e42 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.603122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490758ad-0b22-45d7-85fb-5550439c5e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.612349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 522/1040] 10.222.0.40 () {56 vars in 1133 bytes} [Fri Jul 29 04:35:24 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:24.615325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc6c088-b63f-4073-a633-1d99e9a02047 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:24.622397 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-490758ad-0b22-45d7-85fb-5550439c5e42 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:24.623686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-490758ad-0b22-45d7-85fb-5550439c5e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:24.707835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 519/1041] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:24 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:24.731586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490758ad-0b22-45d7-85fb-5550439c5e42 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:24.732844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc6c088-b63f-4073-a633-1d99e9a02047 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:24.751041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:24.760485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:24.777308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427bb00c-d5bf-4467-aff1-ec506bb60605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:24.777944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427bb00c-d5bf-4467-aff1-ec506bb60605 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.778226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427bb00c-d5bf-4467-aff1-ec506bb60605 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.778507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427bb00c-d5bf-4467-aff1-ec506bb60605 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.852057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427bb00c-d5bf-4467-aff1-ec506bb60605 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:24.853560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427bb00c-d5bf-4467-aff1-ec506bb60605 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:24.862923 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490758ad-0b22-45d7-85fb-5550439c5e42 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:24.906578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f4dbab-6f23-4e4d-a063-901e0d843374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:24.907472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f4dbab-6f23-4e4d-a063-901e0d843374 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:24.907911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f4dbab-6f23-4e4d-a063-901e0d843374 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:24.908344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f4dbab-6f23-4e4d-a063-901e0d843374 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:24.923130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f4dbab-6f23-4e4d-a063-901e0d843374 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-44f4dbab-6f23-4e4d-a063-901e0d843374 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:24.929861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:24.931007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:24.932067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:24.934063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 520/1042] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:35:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:24.940527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f4dbab-6f23-4e4d-a063-901e0d843374 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:24.964719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 523/1043] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:24 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:35:24.969169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427bb00c-d5bf-4467-aff1-ec506bb60605 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:25.003068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:25.021702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:25.059484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f4dbab-6f23-4e4d-a063-901e0d843374 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:25.089638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427bb00c-d5bf-4467-aff1-ec506bb60605 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:25.101838 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb639e9c-3a59-493e-8c9b-3b7364b5b304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:25.102464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb639e9c-3a59-493e-8c9b-3b7364b5b304 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:25.102747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb639e9c-3a59-493e-8c9b-3b7364b5b304 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:25.103023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb639e9c-3a59-493e-8c9b-3b7364b5b304 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:25.103740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb639e9c-3a59-493e-8c9b-3b7364b5b304 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:25.104051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb639e9c-3a59-493e-8c9b-3b7364b5b304 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:25.105496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb639e9c-3a59-493e-8c9b-3b7364b5b304 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:25.115437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 521/1044] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:35:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:25.121012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb639e9c-3a59-493e-8c9b-3b7364b5b304 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:25.126815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd450cd-87f0-4044-87e4-e08ec5eaa98f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:25.127468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd450cd-87f0-4044-87e4-e08ec5eaa98f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:25.127755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd450cd-87f0-4044-87e4-e08ec5eaa98f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:25.128034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd450cd-87f0-4044-87e4-e08ec5eaa98f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:25.138607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd450cd-87f0-4044-87e4-e08ec5eaa98f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:25.140271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd450cd-87f0-4044-87e4-e08ec5eaa98f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-cdd450cd-87f0-4044-87e4-e08ec5eaa98f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:25.155473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:25.156395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:25.157344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:25.157344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:25.157344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:25.157344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:25.161357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 524/1045] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:25.162364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd450cd-87f0-4044-87e4-e08ec5eaa98f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:25.233806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb639e9c-3a59-493e-8c9b-3b7364b5b304 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:25.242877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:25.271195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd450cd-87f0-4044-87e4-e08ec5eaa98f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:25.279424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b78840-11b1-4cf5-a1c8-b1dfa531ef3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:25.280742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b78840-11b1-4cf5-a1c8-b1dfa531ef3f None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:25.281030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b78840-11b1-4cf5-a1c8-b1dfa531ef3f None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:25.281313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b78840-11b1-4cf5-a1c8-b1dfa531ef3f None admin] PATH_INFO: `/v3/users/9f75454caea84e6bae2f295fbfa79ff2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:25.283502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:25.297339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b78840-11b1-4cf5-a1c8-b1dfa531ef3f None admin] RBAC: Authorizing `identity:get_user(user_id=9f75454caea84e6bae2f295fbfa79ff2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:25.298740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b78840-11b1-4cf5-a1c8-b1dfa531ef3f None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:25.300440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 522/1046] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:25 2022] GET /identity/v3/users/9f75454caea84e6bae2f295fbfa79ff2 => generated 285 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:25.305965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b78840-11b1-4cf5-a1c8-b1dfa531ef3f None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:25.317440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec26f4a-6ad3-467c-add4-572e7df0ab81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:25.318075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec26f4a-6ad3-467c-add4-572e7df0ab81 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:25.318366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec26f4a-6ad3-467c-add4-572e7df0ab81 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:25.318649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec26f4a-6ad3-467c-add4-572e7df0ab81 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:25.319359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec26f4a-6ad3-467c-add4-572e7df0ab81 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:25.319674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec26f4a-6ad3-467c-add4-572e7df0ab81 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:25.321110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec26f4a-6ad3-467c-add4-572e7df0ab81 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:25.335203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 525/1047] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:25.340625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec26f4a-6ad3-467c-add4-572e7df0ab81 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:25.421876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b78840-11b1-4cf5-a1c8-b1dfa531ef3f None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:25.446462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec26f4a-6ad3-467c-add4-572e7df0ab81 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:25.459569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be87d4d-e665-4150-b20c-8756c20c9de3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:25.460861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be87d4d-e665-4150-b20c-8756c20c9de3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:25.461222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be87d4d-e665-4150-b20c-8756c20c9de3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:25.461504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be87d4d-e665-4150-b20c-8756c20c9de3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:25.481496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03400421-86e5-49aa-98fb-2a29b5b05c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:25.482143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03400421-86e5-49aa-98fb-2a29b5b05c72 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:25.482432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03400421-86e5-49aa-98fb-2a29b5b05c72 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:25.482713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03400421-86e5-49aa-98fb-2a29b5b05c72 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:25.485026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:25.491377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03400421-86e5-49aa-98fb-2a29b5b05c72 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:25.493349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03400421-86e5-49aa-98fb-2a29b5b05c72 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-03400421-86e5-49aa-98fb-2a29b5b05c72 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:35:25.506313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:25.507212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:25.508088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:35:25.508088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:35:25.508088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:35:25.508088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:35:25.508088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:35:25.508088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:35:25.508088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:25.508088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:25.508088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 526/1048] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:25.513181 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03400421-86e5-49aa-98fb-2a29b5b05c72 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:25.528541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:25.539833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be87d4d-e665-4150-b20c-8756c20c9de3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:25.541322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be87d4d-e665-4150-b20c-8756c20c9de3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:25.626079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03400421-86e5-49aa-98fb-2a29b5b05c72 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:25.661030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 523/1049] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:25 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 29 04:35:25.662564 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be87d4d-e665-4150-b20c-8756c20c9de3 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:25.672532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51d5cf4-3f0a-4cf9-9199-30266e2f1676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:25.673458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51d5cf4-3f0a-4cf9-9199-30266e2f1676 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:25.673750 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51d5cf4-3f0a-4cf9-9199-30266e2f1676 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:25.674031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51d5cf4-3f0a-4cf9-9199-30266e2f1676 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:25.674820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51d5cf4-3f0a-4cf9-9199-30266e2f1676 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:25.675135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51d5cf4-3f0a-4cf9-9199-30266e2f1676 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:25.676571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51d5cf4-3f0a-4cf9-9199-30266e2f1676 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:25.689739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 527/1050] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:35:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:25.696517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51d5cf4-3f0a-4cf9-9199-30266e2f1676 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:25.760880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:25.769843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:25.779732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be87d4d-e665-4150-b20c-8756c20c9de3 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:25.811036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51d5cf4-3f0a-4cf9-9199-30266e2f1676 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:25.816442 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632e43dd-b9d5-4bcc-93d1-659ad627cd9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:25.817119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632e43dd-b9d5-4bcc-93d1-659ad627cd9e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:25.817410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632e43dd-b9d5-4bcc-93d1-659ad627cd9e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:25.817691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632e43dd-b9d5-4bcc-93d1-659ad627cd9e None admin] PATH_INFO: `/v3/projects/874770c5d6ce4541a76c2a02d503ea69` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:25.818851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632e43dd-b9d5-4bcc-93d1-659ad627cd9e None admin] RBAC: Authorizing `identity:get_project(project_id=874770c5d6ce4541a76c2a02d503ea69)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:25.820290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632e43dd-b9d5-4bcc-93d1-659ad627cd9e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:25.822212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 524/1051] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:35:25 2022] GET /identity/v3/projects/874770c5d6ce4541a76c2a02d503ea69 => generated 311 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:25.828577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632e43dd-b9d5-4bcc-93d1-659ad627cd9e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:25.847385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333a4d3-fa25-46de-bb47-295da1c5e2a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:25.848379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333a4d3-fa25-46de-bb47-295da1c5e2a1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:25.848706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333a4d3-fa25-46de-bb47-295da1c5e2a1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:25.848995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333a4d3-fa25-46de-bb47-295da1c5e2a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:25.849791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3333a4d3-fa25-46de-bb47-295da1c5e2a1 None admin] RBAC: Adding query filter params (role.id=8d6877d388f74e08b7653a068705e759, scope.project.id=441bcfe715d44efea69d556677094857, user.id=58c33a72a871432b88da4f0bf1bc514c) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:25.850106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3333a4d3-fa25-46de-bb47-295da1c5e2a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:25.852294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3333a4d3-fa25-46de-bb47-295da1c5e2a1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:25.860427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 528/1052] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:25 2022] GET /identity/v3/role_assignments?user.id=58c33a72a871432b88da4f0bf1bc514c&scope.project.id=441bcfe715d44efea69d556677094857&role.id=8d6877d388f74e08b7653a068705e759 => generated 259 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:25.937820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632e43dd-b9d5-4bcc-93d1-659ad627cd9e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:25.974118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e672ad-4ce9-48c7-a4e6-9e09aa2a5e88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:25.974752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e672ad-4ce9-48c7-a4e6-9e09aa2a5e88 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:25.975034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e672ad-4ce9-48c7-a4e6-9e09aa2a5e88 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:25.975312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e672ad-4ce9-48c7-a4e6-9e09aa2a5e88 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:25.976139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e672ad-4ce9-48c7-a4e6-9e09aa2a5e88 None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=874770c5d6ce4541a76c2a02d503ea69, user.id=9f75454caea84e6bae2f295fbfa79ff2) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:25.976456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e672ad-4ce9-48c7-a4e6-9e09aa2a5e88 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:25.977934 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e672ad-4ce9-48c7-a4e6-9e09aa2a5e88 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:25.984870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 525/1053] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:25 2022] GET /identity/v3/role_assignments?user.id=9f75454caea84e6bae2f295fbfa79ff2&scope.project.id=874770c5d6ce4541a76c2a02d503ea69&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:26.608552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d9cdd5-a3cc-45c6-88d7-baaf0b187fc6 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:26.609126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d9cdd5-a3cc-45c6-88d7-baaf0b187fc6 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:26.609126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d9cdd5-a3cc-45c6-88d7-baaf0b187fc6 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:26.610481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 529/1054] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:26.615814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976af58-5e05-4152-acbd-28f69c45bde7 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:26.616193 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976af58-5e05-4152-acbd-28f69c45bde7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:26.616380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3976af58-5e05-4152-acbd-28f69c45bde7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:26.634414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-3976af58-5e05-4152-acbd-28f69c45bde7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:26.635627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3976af58-5e05-4152-acbd-28f69c45bde7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:26.655905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:26.708019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 526/1055] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:26 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:26.730025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d9cdd5-a3cc-45c6-88d7-baaf0b187fc6 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:26.835136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d9cdd5-a3cc-45c6-88d7-baaf0b187fc6 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:26.852194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:26.870278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75ae7c4-407a-4e6b-a3ae-61136cd0ca2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:26.870847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75ae7c4-407a-4e6b-a3ae-61136cd0ca2e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:26.871128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75ae7c4-407a-4e6b-a3ae-61136cd0ca2e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:26.871402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75ae7c4-407a-4e6b-a3ae-61136cd0ca2e None admin] PATH_INFO: `/v3/services/image` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:26.872024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75ae7c4-407a-4e6b-a3ae-61136cd0ca2e None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:26.873450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75ae7c4-407a-4e6b-a3ae-61136cd0ca2e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-f75ae7c4-407a-4e6b-a3ae-61136cd0ca2e None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:26.882689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:26.883652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:26.885129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 530/1056] 10.222.0.40 () {56 vars in 1105 bytes} [Fri Jul 29 04:35:26 2022] GET /identity/v3/services/image => generated 86 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:26.890005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3976af58-5e05-4152-acbd-28f69c45bde7 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:26.992912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3976af58-5e05-4152-acbd-28f69c45bde7 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:27.036138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343abd1d-acc3-449f-911d-00adb54b8e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:27.036799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343abd1d-acc3-449f-911d-00adb54b8e02 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:27.037086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343abd1d-acc3-449f-911d-00adb54b8e02 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:27.037362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343abd1d-acc3-449f-911d-00adb54b8e02 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:27.038044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343abd1d-acc3-449f-911d-00adb54b8e02 None admin] RBAC: Adding query filter params (name=image) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:27.038350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343abd1d-acc3-449f-911d-00adb54b8e02 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:27.039717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-343abd1d-acc3-449f-911d-00adb54b8e02 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:27.044798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 527/1057] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:26 2022] GET /identity/v3/services?name=image => generated 122 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:27.051062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75ae7c4-407a-4e6b-a3ae-61136cd0ca2e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:27.156102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75ae7c4-407a-4e6b-a3ae-61136cd0ca2e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:27.189493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e738daad-8df3-45db-a5d8-d6728db74f86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:27.190100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e738daad-8df3-45db-a5d8-d6728db74f86 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:27.190379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e738daad-8df3-45db-a5d8-d6728db74f86 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:27.190654 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e738daad-8df3-45db-a5d8-d6728db74f86 None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:27.191326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e738daad-8df3-45db-a5d8-d6728db74f86 None admin] RBAC: Adding query filter params (type=image) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:27.191628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e738daad-8df3-45db-a5d8-d6728db74f86 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:27.193010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e738daad-8df3-45db-a5d8-d6728db74f86 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:27.197860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 531/1058] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:27 2022] GET /identity/v3/services?type=image => generated 350 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:27.203449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343abd1d-acc3-449f-911d-00adb54b8e02 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:27.222822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:27.304088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343abd1d-acc3-449f-911d-00adb54b8e02 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:27.338372 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d71c8bf-7e69-4cc5-a5f3-756da227db3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:27.338989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d71c8bf-7e69-4cc5-a5f3-756da227db3d None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:27.339272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d71c8bf-7e69-4cc5-a5f3-756da227db3d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:27.339547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d71c8bf-7e69-4cc5-a5f3-756da227db3d None admin] PATH_INFO: `/v3/endpoints` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:27.340243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d71c8bf-7e69-4cc5-a5f3-756da227db3d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:27.341625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d71c8bf-7e69-4cc5-a5f3-756da227db3d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:27.368639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 528/1059] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:35:27 2022] POST /identity/v3/endpoints => generated 327 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:35:27.871388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3177db15-315b-4f05-8f4a-3ab06df5798e None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:27.871925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3177db15-315b-4f05-8f4a-3ab06df5798e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:27.871925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3177db15-315b-4f05-8f4a-3ab06df5798e None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:27.873497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 532/1060] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:27.879087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c0f3bf-1d15-432d-94fb-586ec8a208ca None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:27.879469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c0f3bf-1d15-432d-94fb-586ec8a208ca None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:27.879654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c0f3bf-1d15-432d-94fb-586ec8a208ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:27.897812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a4c0f3bf-1d15-432d-94fb-586ec8a208ca None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:27.899019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a4c0f3bf-1d15-432d-94fb-586ec8a208ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:27.967878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:27.975801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 529/1061] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:27 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:27.979204 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201578df-4ad0-4ffa-b992-32516bd8bfc5 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:27.979497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201578df-4ad0-4ffa-b992-32516bd8bfc5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:27.979776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201578df-4ad0-4ffa-b992-32516bd8bfc5 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:27.981266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 533/1062] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:27.986301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00afc07e-1ae6-4fdb-bc4f-2ac5b8293b43 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:27.986596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00afc07e-1ae6-4fdb-bc4f-2ac5b8293b43 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:27.986878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00afc07e-1ae6-4fdb-bc4f-2ac5b8293b43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:27.998269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201578df-4ad0-4ffa-b992-32516bd8bfc5 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.005366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-00afc07e-1ae6-4fdb-bc4f-2ac5b8293b43 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:28.006559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-00afc07e-1ae6-4fdb-bc4f-2ac5b8293b43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:28.052806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:28.081167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 530/1063] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:27 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:28.103167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00afc07e-1ae6-4fdb-bc4f-2ac5b8293b43 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.114903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201578df-4ad0-4ffa-b992-32516bd8bfc5 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:28.149986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14779a4-7bf3-4add-98aa-34a7798ef82e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:28.150627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14779a4-7bf3-4add-98aa-34a7798ef82e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:28.150907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14779a4-7bf3-4add-98aa-34a7798ef82e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:28.151184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14779a4-7bf3-4add-98aa-34a7798ef82e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:28.161499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14779a4-7bf3-4add-98aa-34a7798ef82e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-d14779a4-7bf3-4add-98aa-34a7798ef82e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:28.166316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:28.167256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:28.168075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 534/1064] 10.222.0.40 () {56 vars in 1120 bytes} [Fri Jul 29 04:35:27 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:28.174076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14779a4-7bf3-4add-98aa-34a7798ef82e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.214158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00afc07e-1ae6-4fdb-bc4f-2ac5b8293b43 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:28.246271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:28.253668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee22386-bbb0-4996-ab45-0d1ad14255cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:28.254315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee22386-bbb0-4996-ab45-0d1ad14255cf None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:28.254793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee22386-bbb0-4996-ab45-0d1ad14255cf None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:28.255002 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee22386-bbb0-4996-ab45-0d1ad14255cf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:28.265848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee22386-bbb0-4996-ab45-0d1ad14255cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-6ee22386-bbb0-4996-ab45-0d1ad14255cf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:28.270672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:28.271631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:28.272497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 531/1065] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:35:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:28.277853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee22386-bbb0-4996-ab45-0d1ad14255cf None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.282764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14779a4-7bf3-4add-98aa-34a7798ef82e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:28.290700 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:28.323539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7e3319-9140-49f3-a7d4-fb2295eabc77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:28.324212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7e3319-9140-49f3-a7d4-fb2295eabc77 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:28.324444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7e3319-9140-49f3-a7d4-fb2295eabc77 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:28.324751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7e3319-9140-49f3-a7d4-fb2295eabc77 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:28.325598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7e3319-9140-49f3-a7d4-fb2295eabc77 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:28.325894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7e3319-9140-49f3-a7d4-fb2295eabc77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:28.328116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7e3319-9140-49f3-a7d4-fb2295eabc77 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:28.341554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 535/1066] 10.222.0.40 () {56 vars in 1115 bytes} [Fri Jul 29 04:35:28 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:28.342457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7e3319-9140-49f3-a7d4-fb2295eabc77 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.379625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee22386-bbb0-4996-ab45-0d1ad14255cf None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:28.413635 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5fbd9-fb8f-4f41-af32-d6944620cae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:28.414253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5fbd9-fb8f-4f41-af32-d6944620cae8 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:28.414546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5fbd9-fb8f-4f41-af32-d6944620cae8 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:28.414814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5fbd9-fb8f-4f41-af32-d6944620cae8 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:28.415522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5fbd9-fb8f-4f41-af32-d6944620cae8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:28.415828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5fbd9-fb8f-4f41-af32-d6944620cae8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:28.417222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5fbd9-fb8f-4f41-af32-d6944620cae8 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:28.428627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 532/1067] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:35:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:28.431977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5fbd9-fb8f-4f41-af32-d6944620cae8 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.453671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7e3319-9140-49f3-a7d4-fb2295eabc77 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:28.487022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:28.491526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875c0042-ad84-4116-95d9-590d25376ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:28.492208 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875c0042-ad84-4116-95d9-590d25376ce7 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:28.492494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875c0042-ad84-4116-95d9-590d25376ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:28.492804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875c0042-ad84-4116-95d9-590d25376ce7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:28.505874 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875c0042-ad84-4116-95d9-590d25376ce7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:28.507314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875c0042-ad84-4116-95d9-590d25376ce7 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-875c0042-ad84-4116-95d9-590d25376ce7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:28.522772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:28.523730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:28.525035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:28.525035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:28.525035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:28.525035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:28.525035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 536/1068] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:28.529535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875c0042-ad84-4116-95d9-590d25376ce7 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.548557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5fbd9-fb8f-4f41-af32-d6944620cae8 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:28.552271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:28.586520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6b4c5f-c5e6-4f7e-8ba0-1c2622a968c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:28.587167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6b4c5f-c5e6-4f7e-8ba0-1c2622a968c5 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:28.587450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6b4c5f-c5e6-4f7e-8ba0-1c2622a968c5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:28.587726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6b4c5f-c5e6-4f7e-8ba0-1c2622a968c5 None admin] PATH_INFO: `/v3/users/9f75454caea84e6bae2f295fbfa79ff2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:28.588972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6b4c5f-c5e6-4f7e-8ba0-1c2622a968c5 None admin] RBAC: Authorizing `identity:get_user(user_id=9f75454caea84e6bae2f295fbfa79ff2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:28.590397 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6b4c5f-c5e6-4f7e-8ba0-1c2622a968c5 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:28.596687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 533/1069] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:28 2022] GET /identity/v3/users/9f75454caea84e6bae2f295fbfa79ff2 => generated 285 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:28.597940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6b4c5f-c5e6-4f7e-8ba0-1c2622a968c5 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.634391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875c0042-ad84-4116-95d9-590d25376ce7 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:28.675729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7abeb8e-4862-4d3c-a78f-8abfa75550ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:28.676352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7abeb8e-4862-4d3c-a78f-8abfa75550ef None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:28.676662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7abeb8e-4862-4d3c-a78f-8abfa75550ef None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:28.676969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7abeb8e-4862-4d3c-a78f-8abfa75550ef None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:28.677664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7abeb8e-4862-4d3c-a78f-8abfa75550ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:28.677968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7abeb8e-4862-4d3c-a78f-8abfa75550ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:28.679335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7abeb8e-4862-4d3c-a78f-8abfa75550ef None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:28.692641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 537/1070] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:28.697508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7abeb8e-4862-4d3c-a78f-8abfa75550ef None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.714820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6b4c5f-c5e6-4f7e-8ba0-1c2622a968c5 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:28.736578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:28.752253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb7e46f-56b9-41e2-bf69-de0a2c2ba35b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:28.753089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb7e46f-56b9-41e2-bf69-de0a2c2ba35b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:28.753241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb7e46f-56b9-41e2-bf69-de0a2c2ba35b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:28.753500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb7e46f-56b9-41e2-bf69-de0a2c2ba35b None admin] PATH_INFO: `/v3/projects/874770c5d6ce4541a76c2a02d503ea69` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:28.754770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb7e46f-56b9-41e2-bf69-de0a2c2ba35b None admin] RBAC: Authorizing `identity:get_project(project_id=874770c5d6ce4541a76c2a02d503ea69)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:28.756450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb7e46f-56b9-41e2-bf69-de0a2c2ba35b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:28.763389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 534/1071] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:35:28 2022] GET /identity/v3/projects/874770c5d6ce4541a76c2a02d503ea69 => generated 311 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:28.764007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb7e46f-56b9-41e2-bf69-de0a2c2ba35b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.786098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:28.807109 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7abeb8e-4862-4d3c-a78f-8abfa75550ef None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:28.843687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344c429d-ba25-4d9b-b3ad-1ec2846c4d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:28.844632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344c429d-ba25-4d9b-b3ad-1ec2846c4d0d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:28.845159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344c429d-ba25-4d9b-b3ad-1ec2846c4d0d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:28.845718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344c429d-ba25-4d9b-b3ad-1ec2846c4d0d None admin] PATH_INFO: `/v3/users/nova` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:28.853603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344c429d-ba25-4d9b-b3ad-1ec2846c4d0d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:28.855580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344c429d-ba25-4d9b-b3ad-1ec2846c4d0d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-344c429d-ba25-4d9b-b3ad-1ec2846c4d0d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:35:28.869284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:28.870238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:28.871209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:28.871209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:28.871209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:35:28.871209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:35:28.871209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:35:28.871209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:35:28.871209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:35:28.871209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:35:28.871209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:35:28.871209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:28.872179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb7e46f-56b9-41e2-bf69-de0a2c2ba35b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:28.873976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 538/1072] 10.222.0.40 () {56 vars in 1093 bytes} [Fri Jul 29 04:35:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:28.879671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344c429d-ba25-4d9b-b3ad-1ec2846c4d0d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:28.915068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d002e3-408b-47c6-bf6c-61cd057ea21d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:28.915751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d002e3-408b-47c6-bf6c-61cd057ea21d None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:28.916040 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d002e3-408b-47c6-bf6c-61cd057ea21d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:28.916324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d002e3-408b-47c6-bf6c-61cd057ea21d None admin] PATH_INFO: `/v3/projects/874770c5d6ce4541a76c2a02d503ea69/users/9f75454caea84e6bae2f295fbfa79ff2/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:28.918356 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d002e3-408b-47c6-bf6c-61cd057ea21d None admin] RBAC: Authorizing `identity:create_grant(project_id=874770c5d6ce4541a76c2a02d503ea69, user_id=9f75454caea84e6bae2f295fbfa79ff2, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:28.919800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d002e3-408b-47c6-bf6c-61cd057ea21d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:28.934495 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 535/1073] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:35:28 2022] PUT /identity/v3/projects/874770c5d6ce4541a76c2a02d503ea69/users/9f75454caea84e6bae2f295fbfa79ff2/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:35:28.988653 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344c429d-ba25-4d9b-b3ad-1ec2846c4d0d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:29.027289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4439817e-4ce5-4a9b-a7f8-61f377fb235d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:29.028212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4439817e-4ce5-4a9b-a7f8-61f377fb235d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:29.028694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4439817e-4ce5-4a9b-a7f8-61f377fb235d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:29.029117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4439817e-4ce5-4a9b-a7f8-61f377fb235d None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:29.030995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4439817e-4ce5-4a9b-a7f8-61f377fb235d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:29.031465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4439817e-4ce5-4a9b-a7f8-61f377fb235d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:29.033196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4439817e-4ce5-4a9b-a7f8-61f377fb235d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:29.044076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:29.055579 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 539/1074] 10.222.0.40 () {56 vars in 1133 bytes} [Fri Jul 29 04:35:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:29.061408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d002e3-408b-47c6-bf6c-61cd057ea21d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:29.119202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:29.173646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d002e3-408b-47c6-bf6c-61cd057ea21d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:29.209185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42daf752-fde3-4555-9066-6e5f434855a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:29.210188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42daf752-fde3-4555-9066-6e5f434855a5 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:29.210720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42daf752-fde3-4555-9066-6e5f434855a5 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:29.211221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42daf752-fde3-4555-9066-6e5f434855a5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:29.222053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42daf752-fde3-4555-9066-6e5f434855a5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:29.223733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42daf752-fde3-4555-9066-6e5f434855a5 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-42daf752-fde3-4555-9066-6e5f434855a5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:29.238202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:29.239180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:29.240072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:29.240072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:29.240072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:29.240072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:29.242229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 536/1075] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:29.247978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4439817e-4ce5-4a9b-a7f8-61f377fb235d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:29.350201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4439817e-4ce5-4a9b-a7f8-61f377fb235d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:29.384469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07661477-5098-470c-9bef-4cefd12f5c64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:29.385119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07661477-5098-470c-9bef-4cefd12f5c64 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:29.385451 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07661477-5098-470c-9bef-4cefd12f5c64 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:29.385733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07661477-5098-470c-9bef-4cefd12f5c64 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:29.386425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07661477-5098-470c-9bef-4cefd12f5c64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:29.386727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07661477-5098-470c-9bef-4cefd12f5c64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:29.388089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07661477-5098-470c-9bef-4cefd12f5c64 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:29.404626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 540/1076] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:29.406468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42daf752-fde3-4555-9066-6e5f434855a5 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:29.408236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82efb87d-b380-4f26-8c7a-5e511d49e025 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:29.408749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82efb87d-b380-4f26-8c7a-5e511d49e025 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:29.409227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82efb87d-b380-4f26-8c7a-5e511d49e025 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:29.410689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 541/1077] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:35:29.415666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04379fd-4905-4a0f-8a7e-726d0b35f604 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:29.415959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04379fd-4905-4a0f-8a7e-726d0b35f604 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:29.416244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04379fd-4905-4a0f-8a7e-726d0b35f604 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:29.434924 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-b04379fd-4905-4a0f-8a7e-726d0b35f604 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:29.436759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b04379fd-4905-4a0f-8a7e-726d0b35f604 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:29.494294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:29.509054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:29.512431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42daf752-fde3-4555-9066-6e5f434855a5 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:29.520874 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 542/1078] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:29 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:29.532380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04379fd-4905-4a0f-8a7e-726d0b35f604 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:29.550668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cad591-6a98-4c22-a0d7-ce518d3300b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:29.551591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cad591-6a98-4c22-a0d7-ce518d3300b9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:29.552017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cad591-6a98-4c22-a0d7-ce518d3300b9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:29.552439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cad591-6a98-4c22-a0d7-ce518d3300b9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:29.559608 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48cad591-6a98-4c22-a0d7-ce518d3300b9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:29.561444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48cad591-6a98-4c22-a0d7-ce518d3300b9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-48cad591-6a98-4c22-a0d7-ce518d3300b9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:35:29.573697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:29.574769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:29.575806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:35:29.575806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:35:29.575806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:35:29.575806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:35:29.575806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:35:29.575806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:35:29.575806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:29.575806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:29.578551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 537/1079] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:29.584444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cad591-6a98-4c22-a0d7-ce518d3300b9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:29.642146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04379fd-4905-4a0f-8a7e-726d0b35f604 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:29.687103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bacc7a7-892d-485e-9c24-ebbcb9afcc71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:29.687741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bacc7a7-892d-485e-9c24-ebbcb9afcc71 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:29.688031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bacc7a7-892d-485e-9c24-ebbcb9afcc71 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:29.688316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bacc7a7-892d-485e-9c24-ebbcb9afcc71 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:29.692910 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cad591-6a98-4c22-a0d7-ce518d3300b9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:29.729348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3043cb42-7100-46ab-a315-8d823e2a462d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:29.729985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3043cb42-7100-46ab-a315-8d823e2a462d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:29.730278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3043cb42-7100-46ab-a315-8d823e2a462d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:29.730559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3043cb42-7100-46ab-a315-8d823e2a462d None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:29.731684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3043cb42-7100-46ab-a315-8d823e2a462d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:29.731999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3043cb42-7100-46ab-a315-8d823e2a462d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:29.733971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3043cb42-7100-46ab-a315-8d823e2a462d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:29.746950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:29.751076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 538/1080] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:35:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:29.752386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3043cb42-7100-46ab-a315-8d823e2a462d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:29.762286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bacc7a7-892d-485e-9c24-ebbcb9afcc71 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:29.764647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bacc7a7-892d-485e-9c24-ebbcb9afcc71 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:29.775101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:29.860237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3043cb42-7100-46ab-a315-8d823e2a462d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:29.876511 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 543/1081] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:29 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 29 04:35:29.880150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bacc7a7-892d-485e-9c24-ebbcb9afcc71 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:29.905361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08cf3f-c174-4b81-93db-62cc1d6edbfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:29.906599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08cf3f-c174-4b81-93db-62cc1d6edbfc None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:29.907063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08cf3f-c174-4b81-93db-62cc1d6edbfc None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:29.907631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f08cf3f-c174-4b81-93db-62cc1d6edbfc None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/58c33a72a871432b88da4f0bf1bc514c/roles/8d6877d388f74e08b7653a068705e759` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:29.910751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f08cf3f-c174-4b81-93db-62cc1d6edbfc None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=58c33a72a871432b88da4f0bf1bc514c, role_id=8d6877d388f74e08b7653a068705e759)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:29.912900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f08cf3f-c174-4b81-93db-62cc1d6edbfc None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:29.931530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 539/1082] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:35:29 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/58c33a72a871432b88da4f0bf1bc514c/roles/8d6877d388f74e08b7653a068705e759 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:35:29.974503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:29.990358 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bacc7a7-892d-485e-9c24-ebbcb9afcc71 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:30.034447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8372b6ef-c1aa-41f6-8a9a-5507adc9e1cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:30.035120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8372b6ef-c1aa-41f6-8a9a-5507adc9e1cf None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:30.035459 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8372b6ef-c1aa-41f6-8a9a-5507adc9e1cf None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:30.035747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8372b6ef-c1aa-41f6-8a9a-5507adc9e1cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:30.048585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8372b6ef-c1aa-41f6-8a9a-5507adc9e1cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:30.050179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8372b6ef-c1aa-41f6-8a9a-5507adc9e1cf None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-8372b6ef-c1aa-41f6-8a9a-5507adc9e1cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:30.068701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:30.069670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:30.070552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:30.070552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:30.070552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:30.070552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:30.070552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 544/1083] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:30.075962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08cf3f-c174-4b81-93db-62cc1d6edbfc None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:30.161819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:30.182894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f08cf3f-c174-4b81-93db-62cc1d6edbfc None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:30.220925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76342af4-62fd-4a53-bd0e-98788dfa5d84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:30.221551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76342af4-62fd-4a53-bd0e-98788dfa5d84 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:30.221840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76342af4-62fd-4a53-bd0e-98788dfa5d84 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:30.222120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76342af4-62fd-4a53-bd0e-98788dfa5d84 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:30.222812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76342af4-62fd-4a53-bd0e-98788dfa5d84 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:30.223121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76342af4-62fd-4a53-bd0e-98788dfa5d84 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:30.224479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76342af4-62fd-4a53-bd0e-98788dfa5d84 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:30.244675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 540/1084] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:30.969778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5ddb0b-81f9-4a6d-bc37-c891420942b4 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:30.970277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5ddb0b-81f9-4a6d-bc37-c891420942b4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:30.970277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5ddb0b-81f9-4a6d-bc37-c891420942b4 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:30.971583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 545/1085] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:30.976735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301fbf3a-10f8-4d37-a46b-c80f2bc37905 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:30.977136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301fbf3a-10f8-4d37-a46b-c80f2bc37905 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:30.977297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301fbf3a-10f8-4d37-a46b-c80f2bc37905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:30.996307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-301fbf3a-10f8-4d37-a46b-c80f2bc37905 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:30.997527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-301fbf3a-10f8-4d37-a46b-c80f2bc37905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:31.068001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 541/1086] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:30 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:31.091570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5ddb0b-81f9-4a6d-bc37-c891420942b4 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:31.195179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5ddb0b-81f9-4a6d-bc37-c891420942b4 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:31.228808 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f02e41-3d94-4d2f-8fed-26070810004f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:31.229373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f02e41-3d94-4d2f-8fed-26070810004f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:31.229660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f02e41-3d94-4d2f-8fed-26070810004f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:31.229937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f02e41-3d94-4d2f-8fed-26070810004f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:31.238242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:31.242108 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f02e41-3d94-4d2f-8fed-26070810004f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-65f02e41-3d94-4d2f-8fed-26070810004f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:31.247036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:31.248161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:31.254479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 546/1087] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:35:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:31.255176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301fbf3a-10f8-4d37-a46b-c80f2bc37905 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:31.321944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:31.352071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301fbf3a-10f8-4d37-a46b-c80f2bc37905 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:31.385844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad3edef-8aa7-4376-a9da-6d1d121f8d97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:31.386467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad3edef-8aa7-4376-a9da-6d1d121f8d97 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:31.386755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad3edef-8aa7-4376-a9da-6d1d121f8d97 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:31.387036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad3edef-8aa7-4376-a9da-6d1d121f8d97 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:31.387753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad3edef-8aa7-4376-a9da-6d1d121f8d97 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:31.388064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad3edef-8aa7-4376-a9da-6d1d121f8d97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:31.389740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ad3edef-8aa7-4376-a9da-6d1d121f8d97 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:31.397729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 542/1088] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:35:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:31.403426 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f02e41-3d94-4d2f-8fed-26070810004f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:31.502294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f02e41-3d94-4d2f-8fed-26070810004f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:31.536033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf4d751-14aa-4df8-8f17-d30bd372b98e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:31.536705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf4d751-14aa-4df8-8f17-d30bd372b98e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:31.536995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf4d751-14aa-4df8-8f17-d30bd372b98e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:31.537275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf4d751-14aa-4df8-8f17-d30bd372b98e None admin] PATH_INFO: `/v3/users/9f75454caea84e6bae2f295fbfa79ff2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:31.538438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf4d751-14aa-4df8-8f17-d30bd372b98e None admin] RBAC: Authorizing `identity:get_user(user_id=9f75454caea84e6bae2f295fbfa79ff2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:31.539908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf4d751-14aa-4df8-8f17-d30bd372b98e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:31.541551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 547/1089] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:31 2022] GET /identity/v3/users/9f75454caea84e6bae2f295fbfa79ff2 => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:31.548052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad3edef-8aa7-4376-a9da-6d1d121f8d97 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:31.646194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ad3edef-8aa7-4376-a9da-6d1d121f8d97 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:31.676547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:31.682285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66a257c-a6f5-4ed6-8588-4c41e6512fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:31.682909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66a257c-a6f5-4ed6-8588-4c41e6512fee None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:31.683190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66a257c-a6f5-4ed6-8588-4c41e6512fee None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:31.683464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66a257c-a6f5-4ed6-8588-4c41e6512fee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:31.748792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66a257c-a6f5-4ed6-8588-4c41e6512fee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:31.750227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66a257c-a6f5-4ed6-8588-4c41e6512fee None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:31.851298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 543/1090] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:31 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:35:31.852338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf4d751-14aa-4df8-8f17-d30bd372b98e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:31.918019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:31.948583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8655afc-0de5-41ec-a4c2-72d2e06e5042 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:31.948919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8655afc-0de5-41ec-a4c2-72d2e06e5042 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:31.949203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8655afc-0de5-41ec-a4c2-72d2e06e5042 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:31.950551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 544/1091] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:31.951930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf4d751-14aa-4df8-8f17-d30bd372b98e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:31.956079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4776231-4f9d-4f4a-88f7-f28e2be49b6a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:31.956360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4776231-4f9d-4f4a-88f7-f28e2be49b6a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:31.956667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4776231-4f9d-4f4a-88f7-f28e2be49b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:31.974624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-b4776231-4f9d-4f4a-88f7-f28e2be49b6a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:31.975805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b4776231-4f9d-4f4a-88f7-f28e2be49b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:31.986931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7663b2ca-5c17-4d03-8969-fa25617f5390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:31.987575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7663b2ca-5c17-4d03-8969-fa25617f5390 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:31.987864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7663b2ca-5c17-4d03-8969-fa25617f5390 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:31.988146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7663b2ca-5c17-4d03-8969-fa25617f5390 None admin] PATH_INFO: `/v3/projects/874770c5d6ce4541a76c2a02d503ea69` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:31.989337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7663b2ca-5c17-4d03-8969-fa25617f5390 None admin] RBAC: Authorizing `identity:get_project(project_id=874770c5d6ce4541a76c2a02d503ea69)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:31.990770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7663b2ca-5c17-4d03-8969-fa25617f5390 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:31.992447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 548/1092] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:35:31 2022] GET /identity/v3/projects/874770c5d6ce4541a76c2a02d503ea69 => generated 311 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:31.998311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7663b2ca-5c17-4d03-8969-fa25617f5390 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:32.006253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:32.049291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 545/1093] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:31 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:35:32.071963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4776231-4f9d-4f4a-88f7-f28e2be49b6a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:32.099900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7663b2ca-5c17-4d03-8969-fa25617f5390 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:32.134498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e84330-1326-4617-abf3-25488daf6d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:32.135360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e84330-1326-4617-abf3-25488daf6d0b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:32.135639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e84330-1326-4617-abf3-25488daf6d0b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:32.135916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e84330-1326-4617-abf3-25488daf6d0b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:32.136719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e84330-1326-4617-abf3-25488daf6d0b None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=874770c5d6ce4541a76c2a02d503ea69, user.id=9f75454caea84e6bae2f295fbfa79ff2) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:32.137035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e84330-1326-4617-abf3-25488daf6d0b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:32.138458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e84330-1326-4617-abf3-25488daf6d0b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:32.141823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:32.145211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 549/1094] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:31 2022] GET /identity/v3/role_assignments?user.id=9f75454caea84e6bae2f295fbfa79ff2&scope.project.id=874770c5d6ce4541a76c2a02d503ea69&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:32.182650 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4776231-4f9d-4f4a-88f7-f28e2be49b6a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:32.220174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d6ab68-7ae1-4dff-97d7-3bd16d48295a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:32.220867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d6ab68-7ae1-4dff-97d7-3bd16d48295a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:32.221152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d6ab68-7ae1-4dff-97d7-3bd16d48295a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:32.221428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d6ab68-7ae1-4dff-97d7-3bd16d48295a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:32.232713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d6ab68-7ae1-4dff-97d7-3bd16d48295a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-82d6ab68-7ae1-4dff-97d7-3bd16d48295a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:32.237758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:32.238734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:32.239632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 546/1095] 10.222.0.40 () {56 vars in 1120 bytes} [Fri Jul 29 04:35:32 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:35:32.245319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e84330-1326-4617-abf3-25488daf6d0b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:32.349218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e84330-1326-4617-abf3-25488daf6d0b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:32.383053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a64ac7-e33d-40f0-832a-52f80480254a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:32.383681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a64ac7-e33d-40f0-832a-52f80480254a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:32.383970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a64ac7-e33d-40f0-832a-52f80480254a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:32.384252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a64ac7-e33d-40f0-832a-52f80480254a None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:32.385065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a64ac7-e33d-40f0-832a-52f80480254a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:32.385327 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a64ac7-e33d-40f0-832a-52f80480254a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:32.386704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a64ac7-e33d-40f0-832a-52f80480254a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:32.396469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 550/1096] 10.222.0.40 () {56 vars in 1115 bytes} [Fri Jul 29 04:35:32 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:32.402108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d6ab68-7ae1-4dff-97d7-3bd16d48295a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:32.427183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:32.505066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d6ab68-7ae1-4dff-97d7-3bd16d48295a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:32.548877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4481de44-de4f-4edd-9e45-d03e2d389707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:32.549629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4481de44-de4f-4edd-9e45-d03e2d389707 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:32.549989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4481de44-de4f-4edd-9e45-d03e2d389707 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:32.550339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4481de44-de4f-4edd-9e45-d03e2d389707 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:32.561272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4481de44-de4f-4edd-9e45-d03e2d389707 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:32.562805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4481de44-de4f-4edd-9e45-d03e2d389707 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-4481de44-de4f-4edd-9e45-d03e2d389707 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:32.577272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:32.578313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:32.579223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:32.579223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:32.579223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:32.579223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:32.579223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 547/1097] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:32.584362 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a64ac7-e33d-40f0-832a-52f80480254a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:32.645505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:32.687075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a64ac7-e33d-40f0-832a-52f80480254a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:32.721465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621b2cf6-a3dd-45ba-901f-7ed22aae0fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:32.722017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621b2cf6-a3dd-45ba-901f-7ed22aae0fb9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:32.722297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621b2cf6-a3dd-45ba-901f-7ed22aae0fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:32.722569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621b2cf6-a3dd-45ba-901f-7ed22aae0fb9 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:32.723258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621b2cf6-a3dd-45ba-901f-7ed22aae0fb9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:32.723561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621b2cf6-a3dd-45ba-901f-7ed22aae0fb9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:32.724950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621b2cf6-a3dd-45ba-901f-7ed22aae0fb9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:32.736343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 551/1098] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:32.742243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4481de44-de4f-4edd-9e45-d03e2d389707 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:32.816072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:32.840702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4481de44-de4f-4edd-9e45-d03e2d389707 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:32.874152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f88f59f-59a3-483e-aa94-a2903db2c3e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:32.874797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f88f59f-59a3-483e-aa94-a2903db2c3e4 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:32.875084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f88f59f-59a3-483e-aa94-a2903db2c3e4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:32.875366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f88f59f-59a3-483e-aa94-a2903db2c3e4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:32.883183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f88f59f-59a3-483e-aa94-a2903db2c3e4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:32.884651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f88f59f-59a3-483e-aa94-a2903db2c3e4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-8f88f59f-59a3-483e-aa94-a2903db2c3e4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:35:32.896304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:32.897277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:32.898259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 548/1099] 10.222.0.40 () {56 vars in 1093 bytes} [Fri Jul 29 04:35:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:32.903259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621b2cf6-a3dd-45ba-901f-7ed22aae0fb9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:33.001261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621b2cf6-a3dd-45ba-901f-7ed22aae0fb9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:33.023038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:33.034986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a31f74f-2dd3-4f49-8ef6-cccd92008a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:33.035608 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a31f74f-2dd3-4f49-8ef6-cccd92008a83 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:33.035897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a31f74f-2dd3-4f49-8ef6-cccd92008a83 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:33.036187 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a31f74f-2dd3-4f49-8ef6-cccd92008a83 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:33.037011 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a31f74f-2dd3-4f49-8ef6-cccd92008a83 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:33.037324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a31f74f-2dd3-4f49-8ef6-cccd92008a83 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:33.038753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a31f74f-2dd3-4f49-8ef6-cccd92008a83 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:33.056664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 552/1100] 10.222.0.40 () {56 vars in 1133 bytes} [Fri Jul 29 04:35:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:33.063436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f88f59f-59a3-483e-aa94-a2903db2c3e4 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:33.161444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f88f59f-59a3-483e-aa94-a2903db2c3e4 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:33.195630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406dc68a-dbc5-4ae3-bfa1-b4f1a5db2bb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:33.196258 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406dc68a-dbc5-4ae3-bfa1-b4f1a5db2bb1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:33.196549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406dc68a-dbc5-4ae3-bfa1-b4f1a5db2bb1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:33.196856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406dc68a-dbc5-4ae3-bfa1-b4f1a5db2bb1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:33.199454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:33.266190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406dc68a-dbc5-4ae3-bfa1-b4f1a5db2bb1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:33.267662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-406dc68a-dbc5-4ae3-bfa1-b4f1a5db2bb1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:33.368015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 549/1101] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:33 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:35:33.374478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a31f74f-2dd3-4f49-8ef6-cccd92008a83 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:33.473211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a31f74f-2dd3-4f49-8ef6-cccd92008a83 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:33.505796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78a565-51eb-4e10-96a2-85477c5cbdb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:33.506433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78a565-51eb-4e10-96a2-85477c5cbdb5 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:33.506724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78a565-51eb-4e10-96a2-85477c5cbdb5 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:33.507006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78a565-51eb-4e10-96a2-85477c5cbdb5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:33.517260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a78a565-51eb-4e10-96a2-85477c5cbdb5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:33.518698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a78a565-51eb-4e10-96a2-85477c5cbdb5 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-8a78a565-51eb-4e10-96a2-85477c5cbdb5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:35:33.533550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:33.534682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:33.535628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:35:33.535628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:35:33.535628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:35:33.535628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:33.535628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 553/1102] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:33.540288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406dc68a-dbc5-4ae3-bfa1-b4f1a5db2bb1 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:33.572994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:33.640055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406dc68a-dbc5-4ae3-bfa1-b4f1a5db2bb1 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:33.678825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f229b88-5363-4c4e-8e9f-53f95369e7ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:33.679453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f229b88-5363-4c4e-8e9f-53f95369e7ca None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:33.679743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f229b88-5363-4c4e-8e9f-53f95369e7ca None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:33.680029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f229b88-5363-4c4e-8e9f-53f95369e7ca None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:33.680762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f229b88-5363-4c4e-8e9f-53f95369e7ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:33.681076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f229b88-5363-4c4e-8e9f-53f95369e7ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:33.682457 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f229b88-5363-4c4e-8e9f-53f95369e7ca None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:33.694190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 550/1103] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:35:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:35:33.699805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78a565-51eb-4e10-96a2-85477c5cbdb5 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:33.731423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:33.800403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78a565-51eb-4e10-96a2-85477c5cbdb5 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:33.832823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4d253e-7033-46b4-9826-fd09e4747552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:33.833468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4d253e-7033-46b4-9826-fd09e4747552 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:33.833758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4d253e-7033-46b4-9826-fd09e4747552 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:33.834041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4d253e-7033-46b4-9826-fd09e4747552 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:33.840574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4d253e-7033-46b4-9826-fd09e4747552 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:33.842634 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4d253e-7033-46b4-9826-fd09e4747552 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-cb4d253e-7033-46b4-9826-fd09e4747552 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:35:33.854073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:33.855091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:33.856382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:35:33.856382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:35:33.856382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:35:33.856382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:35:33.856382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:35:33.856382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:35:33.856382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:35:33.856382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:33.856382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 554/1104] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:35:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:33.861320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f229b88-5363-4c4e-8e9f-53f95369e7ca None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:33.957274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:33.964632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f229b88-5363-4c4e-8e9f-53f95369e7ca None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:33.997726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae9f677-1d0c-44c8-97fd-b45cc26db513 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:33.998815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae9f677-1d0c-44c8-97fd-b45cc26db513 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:33.999126 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae9f677-1d0c-44c8-97fd-b45cc26db513 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:33.999389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae9f677-1d0c-44c8-97fd-b45cc26db513 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:34.000140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae9f677-1d0c-44c8-97fd-b45cc26db513 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:34.000456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae9f677-1d0c-44c8-97fd-b45cc26db513 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:34.001931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae9f677-1d0c-44c8-97fd-b45cc26db513 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:34.013713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 551/1105] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:35:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:34.019805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4d253e-7033-46b4-9826-fd09e4747552 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:34.114415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:34.123597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4d253e-7033-46b4-9826-fd09e4747552 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:34.157475 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da72ef36-b14f-4bdc-a786-1bb88b3cc77b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:34.158088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72ef36-b14f-4bdc-a786-1bb88b3cc77b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:34.158397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72ef36-b14f-4bdc-a786-1bb88b3cc77b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:34.158671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da72ef36-b14f-4bdc-a786-1bb88b3cc77b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:34.159449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da72ef36-b14f-4bdc-a786-1bb88b3cc77b None admin] RBAC: Adding query filter params (role.id=8d6877d388f74e08b7653a068705e759, scope.project.id=441bcfe715d44efea69d556677094857, user.id=58c33a72a871432b88da4f0bf1bc514c) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:34.159755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da72ef36-b14f-4bdc-a786-1bb88b3cc77b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:34.161233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da72ef36-b14f-4bdc-a786-1bb88b3cc77b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:34.167060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 555/1106] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:34 2022] GET /identity/v3/role_assignments?user.id=58c33a72a871432b88da4f0bf1bc514c&scope.project.id=441bcfe715d44efea69d556677094857&role.id=8d6877d388f74e08b7653a068705e759 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:34.189791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d515913b-6f82-4c4d-82c5-af896f14af6b None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:34.190328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d515913b-6f82-4c4d-82c5-af896f14af6b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:34.190451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d515913b-6f82-4c4d-82c5-af896f14af6b None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:34.191811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 552/1107] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:34.197714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf3a04a-25cf-4b74-a6e6-6b9e945fbb39 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:34.198004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf3a04a-25cf-4b74-a6e6-6b9e945fbb39 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:34.198284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf3a04a-25cf-4b74-a6e6-6b9e945fbb39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:34.217963 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-8cf3a04a-25cf-4b74-a6e6-6b9e945fbb39 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:34.219210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8cf3a04a-25cf-4b74-a6e6-6b9e945fbb39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:34.291986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 556/1108] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:34 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:34.317302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d515913b-6f82-4c4d-82c5-af896f14af6b None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:34.419431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d515913b-6f82-4c4d-82c5-af896f14af6b None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:34.454645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd9d523-094d-4926-806a-88030977837b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:34.455282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd9d523-094d-4926-806a-88030977837b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:34.455565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd9d523-094d-4926-806a-88030977837b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:34.455848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd9d523-094d-4926-806a-88030977837b None admin] PATH_INFO: `/v3/domains/default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:34.457027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd9d523-094d-4926-806a-88030977837b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:34.458459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd9d523-094d-4926-806a-88030977837b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:34.460041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 553/1109] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:34.465931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf3a04a-25cf-4b74-a6e6-6b9e945fbb39 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:34.553218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:34.568965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf3a04a-25cf-4b74-a6e6-6b9e945fbb39 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:34.604022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4964b6-0cad-4b81-b3d0-a6253d6a2818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:34.604675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4964b6-0cad-4b81-b3d0-a6253d6a2818 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:34.604968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4964b6-0cad-4b81-b3d0-a6253d6a2818 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:34.605250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4964b6-0cad-4b81-b3d0-a6253d6a2818 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:34.605957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4964b6-0cad-4b81-b3d0-a6253d6a2818 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:34.607389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4964b6-0cad-4b81-b3d0-a6253d6a2818 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:34.630190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 557/1110] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:35:34 2022] POST /identity/v3/users => generated 286 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:35:36.687769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09529071-8976-4067-b911-f5bf3ddd41a0 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:36.688250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09529071-8976-4067-b911-f5bf3ddd41a0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:36.688366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09529071-8976-4067-b911-f5bf3ddd41a0 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:36.689720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 554/1111] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:36.694873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37a11b1-2dc8-44d5-bbae-6df803008fe7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:36.695258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37a11b1-2dc8-44d5-bbae-6df803008fe7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:36.695435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37a11b1-2dc8-44d5-bbae-6df803008fe7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:36.714289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e37a11b1-2dc8-44d5-bbae-6df803008fe7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:36.715474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e37a11b1-2dc8-44d5-bbae-6df803008fe7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:36.780177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 558/1112] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:36 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:36.803972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09529071-8976-4067-b911-f5bf3ddd41a0 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:36.827965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:36.909187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09529071-8976-4067-b911-f5bf3ddd41a0 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:36.949097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac02d13e-c498-4dc7-88c1-d19c3b419c7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:36.949728 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac02d13e-c498-4dc7-88c1-d19c3b419c7e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:36.950019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac02d13e-c498-4dc7-88c1-d19c3b419c7e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:36.950302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac02d13e-c498-4dc7-88c1-d19c3b419c7e None admin] PATH_INFO: `/v3/roles/28ddf4e7cb7f4139a3b6a79077418921` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:36.951770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac02d13e-c498-4dc7-88c1-d19c3b419c7e None admin] RBAC: Authorizing `identity:get_role(role_id=28ddf4e7cb7f4139a3b6a79077418921)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:36.953243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac02d13e-c498-4dc7-88c1-d19c3b419c7e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:36.954587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 555/1113] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:36 2022] GET /identity/v3/roles/28ddf4e7cb7f4139a3b6a79077418921 => generated 223 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:36.960356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37a11b1-2dc8-44d5-bbae-6df803008fe7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:37.027290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:37.057774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37a11b1-2dc8-44d5-bbae-6df803008fe7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:37.091541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88aa981-f5f3-41f2-a478-23a704951616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:37.092832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88aa981-f5f3-41f2-a478-23a704951616 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:37.093124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88aa981-f5f3-41f2-a478-23a704951616 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:37.093404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e88aa981-f5f3-41f2-a478-23a704951616 None admin] PATH_INFO: `/v3/users/785f9770496342bf9cc4dc28e0d41947` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:37.106475 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88aa981-f5f3-41f2-a478-23a704951616 None admin] RBAC: Authorizing `identity:get_user(user_id=785f9770496342bf9cc4dc28e0d41947)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:37.108228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e88aa981-f5f3-41f2-a478-23a704951616 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:37.109899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 559/1114] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:36 2022] GET /identity/v3/users/785f9770496342bf9cc4dc28e0d41947 => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:37.115653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac02d13e-c498-4dc7-88c1-d19c3b419c7e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:37.203982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:37.213338 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac02d13e-c498-4dc7-88c1-d19c3b419c7e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:37.248065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e01e81c-b5f9-4d07-a6ca-5358def1999b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:37.248892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e01e81c-b5f9-4d07-a6ca-5358def1999b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:37.249181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e01e81c-b5f9-4d07-a6ca-5358def1999b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:37.249507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e01e81c-b5f9-4d07-a6ca-5358def1999b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:37.315412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e01e81c-b5f9-4d07-a6ca-5358def1999b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:37.316897 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e01e81c-b5f9-4d07-a6ca-5358def1999b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:37.403100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:37.413944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 556/1115] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:37 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:35:37.420150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88aa981-f5f3-41f2-a478-23a704951616 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:37.517611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e88aa981-f5f3-41f2-a478-23a704951616 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:37.545253 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:37.551400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e70332c-bb32-4f25-82f4-0aee9a5889e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:37.552034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e70332c-bb32-4f25-82f4-0aee9a5889e1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:37.552319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e70332c-bb32-4f25-82f4-0aee9a5889e1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:37.552619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e70332c-bb32-4f25-82f4-0aee9a5889e1 None admin] PATH_INFO: `/v3/projects/874770c5d6ce4541a76c2a02d503ea69` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:37.553770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e70332c-bb32-4f25-82f4-0aee9a5889e1 None admin] RBAC: Authorizing `identity:get_project(project_id=874770c5d6ce4541a76c2a02d503ea69)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:37.555195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e70332c-bb32-4f25-82f4-0aee9a5889e1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:37.556872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 560/1116] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:35:37 2022] GET /identity/v3/projects/874770c5d6ce4541a76c2a02d503ea69 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:37.563547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e01e81c-b5f9-4d07-a6ca-5358def1999b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:37.659311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e01e81c-b5f9-4d07-a6ca-5358def1999b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:37.692837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0ce6a0-1263-42ed-bc3b-527f0171e23b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:37.693449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0ce6a0-1263-42ed-bc3b-527f0171e23b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:37.693711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0ce6a0-1263-42ed-bc3b-527f0171e23b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:37.693982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0ce6a0-1263-42ed-bc3b-527f0171e23b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:37.694767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0ce6a0-1263-42ed-bc3b-527f0171e23b None admin] RBAC: Adding query filter params (role.id=28ddf4e7cb7f4139a3b6a79077418921, scope.project.id=874770c5d6ce4541a76c2a02d503ea69, user.id=785f9770496342bf9cc4dc28e0d41947) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:37.695076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0ce6a0-1263-42ed-bc3b-527f0171e23b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:37.696505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0ce6a0-1263-42ed-bc3b-527f0171e23b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:37.702557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 557/1117] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:37 2022] GET /identity/v3/role_assignments?user.id=785f9770496342bf9cc4dc28e0d41947&scope.project.id=874770c5d6ce4541a76c2a02d503ea69&role.id=28ddf4e7cb7f4139a3b6a79077418921 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:39.656269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144baab-070d-4cd7-a18e-b82daf1ad12c None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:39.656733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144baab-070d-4cd7-a18e-b82daf1ad12c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:39.656879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c144baab-070d-4cd7-a18e-b82daf1ad12c None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:39.658236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 561/1118] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:39.663324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304d6bba-649a-4aff-b50d-7c5c4be02ae8 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:39.663787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304d6bba-649a-4aff-b50d-7c5c4be02ae8 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:39.663890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304d6bba-649a-4aff-b50d-7c5c4be02ae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:39.681929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-304d6bba-649a-4aff-b50d-7c5c4be02ae8 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:39.683129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-304d6bba-649a-4aff-b50d-7c5c4be02ae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:39.731915 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:39.749877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 558/1119] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:39 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:39.772256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c144baab-070d-4cd7-a18e-b82daf1ad12c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:39.869728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c144baab-070d-4cd7-a18e-b82daf1ad12c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:39.903255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d77967-c7f3-4813-b595-7bb3e680e47e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:39.903824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d77967-c7f3-4813-b595-7bb3e680e47e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:39.904134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d77967-c7f3-4813-b595-7bb3e680e47e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:39.904421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d77967-c7f3-4813-b595-7bb3e680e47e None admin] PATH_INFO: `/v3/roles/28ddf4e7cb7f4139a3b6a79077418921` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:39.905892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d77967-c7f3-4813-b595-7bb3e680e47e None admin] RBAC: Authorizing `identity:get_role(role_id=28ddf4e7cb7f4139a3b6a79077418921)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:39.907255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d77967-c7f3-4813-b595-7bb3e680e47e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:39.908572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 562/1120] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:39 2022] GET /identity/v3/roles/28ddf4e7cb7f4139a3b6a79077418921 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:39.914375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304d6bba-649a-4aff-b50d-7c5c4be02ae8 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:40.011334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304d6bba-649a-4aff-b50d-7c5c4be02ae8 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:40.044317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cf17f1-773a-48d5-9e53-eb72738f00e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:40.045295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cf17f1-773a-48d5-9e53-eb72738f00e9 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:40.045587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cf17f1-773a-48d5-9e53-eb72738f00e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:40.045868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cf17f1-773a-48d5-9e53-eb72738f00e9 None admin] PATH_INFO: `/v3/users/785f9770496342bf9cc4dc28e0d41947` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:40.047043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cf17f1-773a-48d5-9e53-eb72738f00e9 None admin] RBAC: Authorizing `identity:get_user(user_id=785f9770496342bf9cc4dc28e0d41947)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:40.048754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cf17f1-773a-48d5-9e53-eb72738f00e9 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:40.050350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 559/1121] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:39 2022] GET /identity/v3/users/785f9770496342bf9cc4dc28e0d41947 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:40.056038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d77967-c7f3-4813-b595-7bb3e680e47e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:40.120815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:40.156178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d77967-c7f3-4813-b595-7bb3e680e47e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:40.189572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d71335-2886-4e34-811b-7b24cb346b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:40.190202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d71335-2886-4e34-811b-7b24cb346b30 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:40.190482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d71335-2886-4e34-811b-7b24cb346b30 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:40.190759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d71335-2886-4e34-811b-7b24cb346b30 None admin] PATH_INFO: `/v3/projects/874770c5d6ce4541a76c2a02d503ea69` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:40.191888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d71335-2886-4e34-811b-7b24cb346b30 None admin] RBAC: Authorizing `identity:get_project(project_id=874770c5d6ce4541a76c2a02d503ea69)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:40.193345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d71335-2886-4e34-811b-7b24cb346b30 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:40.194970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 563/1122] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:35:40 2022] GET /identity/v3/projects/874770c5d6ce4541a76c2a02d503ea69 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:40.200893 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cf17f1-773a-48d5-9e53-eb72738f00e9 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:40.253200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:40.299345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cf17f1-773a-48d5-9e53-eb72738f00e9 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:40.332537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1294c2-35c9-46a1-8ad3-07a941d8e030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:40.333219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1294c2-35c9-46a1-8ad3-07a941d8e030 None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:40.333509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1294c2-35c9-46a1-8ad3-07a941d8e030 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:40.333785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1294c2-35c9-46a1-8ad3-07a941d8e030 None admin] PATH_INFO: `/v3/projects/874770c5d6ce4541a76c2a02d503ea69/users/785f9770496342bf9cc4dc28e0d41947/roles/28ddf4e7cb7f4139a3b6a79077418921` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:40.335968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1294c2-35c9-46a1-8ad3-07a941d8e030 None admin] RBAC: Authorizing `identity:create_grant(project_id=874770c5d6ce4541a76c2a02d503ea69, user_id=785f9770496342bf9cc4dc28e0d41947, role_id=28ddf4e7cb7f4139a3b6a79077418921)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:40.337424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1294c2-35c9-46a1-8ad3-07a941d8e030 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:40.349721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 560/1123] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:35:40 2022] PUT /identity/v3/projects/874770c5d6ce4541a76c2a02d503ea69/users/785f9770496342bf9cc4dc28e0d41947/roles/28ddf4e7cb7f4139a3b6a79077418921 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:35:42.340041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184a43f-0422-4c3a-abe9-bcb1f3100f10 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:42.340579 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184a43f-0422-4c3a-abe9-bcb1f3100f10 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:42.340724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6184a43f-0422-4c3a-abe9-bcb1f3100f10 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:42.342113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 564/1124] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:42.347504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d20aa94-3375-4455-a225-77ba56da0487 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:42.347930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d20aa94-3375-4455-a225-77ba56da0487 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:42.348075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d20aa94-3375-4455-a225-77ba56da0487 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:42.367615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7d20aa94-3375-4455-a225-77ba56da0487 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:42.369290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7d20aa94-3375-4455-a225-77ba56da0487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:42.435116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 561/1125] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:42 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:42.459960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6184a43f-0422-4c3a-abe9-bcb1f3100f10 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:42.561611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6184a43f-0422-4c3a-abe9-bcb1f3100f10 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:42.594138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9b65c5-97c2-447a-a0ef-48aee43ac3f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:42.594704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9b65c5-97c2-447a-a0ef-48aee43ac3f9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:42.594991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9b65c5-97c2-447a-a0ef-48aee43ac3f9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:42.595273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9b65c5-97c2-447a-a0ef-48aee43ac3f9 None admin] PATH_INFO: `/v3/roles/28ddf4e7cb7f4139a3b6a79077418921` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:42.596743 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9b65c5-97c2-447a-a0ef-48aee43ac3f9 None admin] RBAC: Authorizing `identity:get_role(role_id=28ddf4e7cb7f4139a3b6a79077418921)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:42.598121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9b65c5-97c2-447a-a0ef-48aee43ac3f9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:42.599451 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 565/1126] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:42 2022] GET /identity/v3/roles/28ddf4e7cb7f4139a3b6a79077418921 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:42.605344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d20aa94-3375-4455-a225-77ba56da0487 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:42.650417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:42.709882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d20aa94-3375-4455-a225-77ba56da0487 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:42.743907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffe3a68-fa68-4098-b4ba-226142deab37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:42.744549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffe3a68-fa68-4098-b4ba-226142deab37 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:42.744868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffe3a68-fa68-4098-b4ba-226142deab37 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:42.745151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffe3a68-fa68-4098-b4ba-226142deab37 None admin] PATH_INFO: `/v3/users/785f9770496342bf9cc4dc28e0d41947` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:42.746313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffe3a68-fa68-4098-b4ba-226142deab37 None admin] RBAC: Authorizing `identity:get_user(user_id=785f9770496342bf9cc4dc28e0d41947)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:42.747775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ffe3a68-fa68-4098-b4ba-226142deab37 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:42.749429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 562/1127] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:42 2022] GET /identity/v3/users/785f9770496342bf9cc4dc28e0d41947 => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:42.757054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9b65c5-97c2-447a-a0ef-48aee43ac3f9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:42.760757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:42.885930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9b65c5-97c2-447a-a0ef-48aee43ac3f9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:42.930207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435f8e22-77cb-4350-996d-c9f5720ca7bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:42.930984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435f8e22-77cb-4350-996d-c9f5720ca7bf None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:42.931382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435f8e22-77cb-4350-996d-c9f5720ca7bf None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:42.931773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435f8e22-77cb-4350-996d-c9f5720ca7bf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:43.016951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435f8e22-77cb-4350-996d-c9f5720ca7bf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:43.018980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435f8e22-77cb-4350-996d-c9f5720ca7bf None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:43.021349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:43.123998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 566/1128] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:42 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:35:43.129944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffe3a68-fa68-4098-b4ba-226142deab37 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:43.228738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ffe3a68-fa68-4098-b4ba-226142deab37 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:43.239712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:43.264001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c040ee-b05e-4aa6-8e36-918c2a517c96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:43.264718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c040ee-b05e-4aa6-8e36-918c2a517c96 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:43.265022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c040ee-b05e-4aa6-8e36-918c2a517c96 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:43.265314 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c040ee-b05e-4aa6-8e36-918c2a517c96 None admin] PATH_INFO: `/v3/projects/874770c5d6ce4541a76c2a02d503ea69` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:43.266532 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c040ee-b05e-4aa6-8e36-918c2a517c96 None admin] RBAC: Authorizing `identity:get_project(project_id=874770c5d6ce4541a76c2a02d503ea69)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:43.268052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c040ee-b05e-4aa6-8e36-918c2a517c96 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:43.269865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 563/1129] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:35:43 2022] GET /identity/v3/projects/874770c5d6ce4541a76c2a02d503ea69 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:43.276017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435f8e22-77cb-4350-996d-c9f5720ca7bf None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:43.377764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435f8e22-77cb-4350-996d-c9f5720ca7bf None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:43.379345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:43.414311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9e3980-640c-4819-a798-82cd937bba34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:43.414959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9e3980-640c-4819-a798-82cd937bba34 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:43.415264 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9e3980-640c-4819-a798-82cd937bba34 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:43.415560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9e3980-640c-4819-a798-82cd937bba34 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:43.416392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9e3980-640c-4819-a798-82cd937bba34 None admin] RBAC: Adding query filter params (role.id=28ddf4e7cb7f4139a3b6a79077418921, scope.project.id=874770c5d6ce4541a76c2a02d503ea69, user.id=785f9770496342bf9cc4dc28e0d41947) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:43.416751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9e3980-640c-4819-a798-82cd937bba34 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:43.418286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9e3980-640c-4819-a798-82cd937bba34 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:43.424545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 567/1130] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:43 2022] GET /identity/v3/role_assignments?user.id=785f9770496342bf9cc4dc28e0d41947&scope.project.id=874770c5d6ce4541a76c2a02d503ea69&role.id=28ddf4e7cb7f4139a3b6a79077418921 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:45.521925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9353104-8d37-44d1-b0ee-1a7e67077cbc None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:45.522466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9353104-8d37-44d1-b0ee-1a7e67077cbc None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:45.522670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9353104-8d37-44d1-b0ee-1a7e67077cbc None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:45.524045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 564/1131] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:45.530044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7942994b-4c2b-4398-a04b-147eca4715cf None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:45.530492 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7942994b-4c2b-4398-a04b-147eca4715cf None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:45.530603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7942994b-4c2b-4398-a04b-147eca4715cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:45.548711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-7942994b-4c2b-4398-a04b-147eca4715cf None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:45.549912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7942994b-4c2b-4398-a04b-147eca4715cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:45.614775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 568/1132] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:45 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:45.637303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9353104-8d37-44d1-b0ee-1a7e67077cbc None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:45.738973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9353104-8d37-44d1-b0ee-1a7e67077cbc None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:45.773011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3cf23d-1059-4a8e-b7a1-eeeca142afbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:45.773583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3cf23d-1059-4a8e-b7a1-eeeca142afbd None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:45.773870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3cf23d-1059-4a8e-b7a1-eeeca142afbd None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:45.774152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3cf23d-1059-4a8e-b7a1-eeeca142afbd None admin] PATH_INFO: `/v3/domains/default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:45.775315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3cf23d-1059-4a8e-b7a1-eeeca142afbd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:45.776783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3cf23d-1059-4a8e-b7a1-eeeca142afbd None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:45.778384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 565/1133] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:45.784230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7942994b-4c2b-4398-a04b-147eca4715cf None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:45.875267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:45.881305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7942994b-4c2b-4398-a04b-147eca4715cf None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:45.914694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54158ed3-e9a0-40fd-855a-f1a01f157f17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:45.915305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54158ed3-e9a0-40fd-855a-f1a01f157f17 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:45.915595 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54158ed3-e9a0-40fd-855a-f1a01f157f17 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:45.915878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54158ed3-e9a0-40fd-855a-f1a01f157f17 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:45.916577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54158ed3-e9a0-40fd-855a-f1a01f157f17 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:45.918043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54158ed3-e9a0-40fd-855a-f1a01f157f17 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:45.944560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 569/1134] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:35:45 2022] POST /identity/v3/projects => generated 311 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:35:48.012762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d98176-f55e-4ce5-8777-3f36a528a8b0 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:48.013914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d98176-f55e-4ce5-8777-3f36a528a8b0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:48.014084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5d98176-f55e-4ce5-8777-3f36a528a8b0 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:48.015456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 566/1135] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:48.020758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a033972d-89df-4d39-9b9c-c85873df61ac None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:48.021131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a033972d-89df-4d39-9b9c-c85873df61ac None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:48.021328 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a033972d-89df-4d39-9b9c-c85873df61ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:48.039310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-a033972d-89df-4d39-9b9c-c85873df61ac None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:48.040565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a033972d-89df-4d39-9b9c-c85873df61ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:48.105792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 570/1136] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:48 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:48.130572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d98176-f55e-4ce5-8777-3f36a528a8b0 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:48.177972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:48.232681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5d98176-f55e-4ce5-8777-3f36a528a8b0 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:48.267210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e11648-bf40-42fe-b218-b56274fc3f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:48.267785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e11648-bf40-42fe-b218-b56274fc3f32 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:48.268074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e11648-bf40-42fe-b218-b56274fc3f32 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:48.268356 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e11648-bf40-42fe-b218-b56274fc3f32 None admin] PATH_INFO: `/v3/domains/default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:48.269546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e11648-bf40-42fe-b218-b56274fc3f32 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:48.270980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e11648-bf40-42fe-b218-b56274fc3f32 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:48.272589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 567/1137] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:35:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:48.278482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a033972d-89df-4d39-9b9c-c85873df61ac None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:48.353470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:48.376044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a033972d-89df-4d39-9b9c-c85873df61ac None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:48.408568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65ac2a0-6e9c-422d-b6e8-b8537fccd6e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:48.409209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65ac2a0-6e9c-422d-b6e8-b8537fccd6e7 None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:48.409495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65ac2a0-6e9c-422d-b6e8-b8537fccd6e7 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:48.409773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65ac2a0-6e9c-422d-b6e8-b8537fccd6e7 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:48.410449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65ac2a0-6e9c-422d-b6e8-b8537fccd6e7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:48.411881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65ac2a0-6e9c-422d-b6e8-b8537fccd6e7 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:48.432545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 571/1138] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:35:48 2022] POST /identity/v3/users => generated 286 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:35:50.439522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6ac3c6-af2f-41a2-adbe-eef47d70cb21 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:50.440109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6ac3c6-af2f-41a2-adbe-eef47d70cb21 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:50.440109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6ac3c6-af2f-41a2-adbe-eef47d70cb21 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:50.441468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 568/1139] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:50.446530 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cba468-53e3-47f9-8f66-34fad41e5e1b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:50.446948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cba468-53e3-47f9-8f66-34fad41e5e1b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:50.447095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cba468-53e3-47f9-8f66-34fad41e5e1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:50.465065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-b5cba468-53e3-47f9-8f66-34fad41e5e1b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:50.467052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b5cba468-53e3-47f9-8f66-34fad41e5e1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:50.533569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 572/1140] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:50 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:50.557536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6ac3c6-af2f-41a2-adbe-eef47d70cb21 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:50.658066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6ac3c6-af2f-41a2-adbe-eef47d70cb21 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:50.672787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:50.693024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d359f8-b967-4190-b1be-2d6580fa4bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:50.693610 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d359f8-b967-4190-b1be-2d6580fa4bad None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:50.693897 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d359f8-b967-4190-b1be-2d6580fa4bad None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:50.694178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d359f8-b967-4190-b1be-2d6580fa4bad None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:50.704865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d359f8-b967-4190-b1be-2d6580fa4bad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-b3d359f8-b967-4190-b1be-2d6580fa4bad None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:50.709703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:50.710924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:50.712194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 569/1141] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:35:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:50.718053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cba468-53e3-47f9-8f66-34fad41e5e1b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:50.780436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:50.818519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cba468-53e3-47f9-8f66-34fad41e5e1b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:50.852321 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885d5af8-e8df-4e4a-9368-4186f1f1c73f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:50.852957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885d5af8-e8df-4e4a-9368-4186f1f1c73f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:50.853244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885d5af8-e8df-4e4a-9368-4186f1f1c73f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:50.853520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885d5af8-e8df-4e4a-9368-4186f1f1c73f None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:50.854230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885d5af8-e8df-4e4a-9368-4186f1f1c73f None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:50.854535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885d5af8-e8df-4e4a-9368-4186f1f1c73f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:50.855893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885d5af8-e8df-4e4a-9368-4186f1f1c73f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:50.864119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 573/1142] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:35:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:50.869817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d359f8-b967-4190-b1be-2d6580fa4bad None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:50.969469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d359f8-b967-4190-b1be-2d6580fa4bad None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:51.002416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2c08d7-c3f5-4099-add9-d89359cf697b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:51.003037 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2c08d7-c3f5-4099-add9-d89359cf697b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:51.003324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2c08d7-c3f5-4099-add9-d89359cf697b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:51.003600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2c08d7-c3f5-4099-add9-d89359cf697b None admin] PATH_INFO: `/v3/users/2cbbcf61163049e293aa90eaa3aa0ce2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:51.016741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2c08d7-c3f5-4099-add9-d89359cf697b None admin] RBAC: Authorizing `identity:get_user(user_id=2cbbcf61163049e293aa90eaa3aa0ce2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:51.018168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2c08d7-c3f5-4099-add9-d89359cf697b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:51.019748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 570/1143] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:50 2022] GET /identity/v3/users/2cbbcf61163049e293aa90eaa3aa0ce2 => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:51.025398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885d5af8-e8df-4e4a-9368-4186f1f1c73f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:51.121291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885d5af8-e8df-4e4a-9368-4186f1f1c73f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:51.147409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:51.156353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd7f979-7d0f-4c6d-9bf3-1c628fa8ae0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:51.156995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd7f979-7d0f-4c6d-9bf3-1c628fa8ae0d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:51.157281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd7f979-7d0f-4c6d-9bf3-1c628fa8ae0d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:51.157556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd7f979-7d0f-4c6d-9bf3-1c628fa8ae0d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:51.222991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd7f979-7d0f-4c6d-9bf3-1c628fa8ae0d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:51.224365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd7f979-7d0f-4c6d-9bf3-1c628fa8ae0d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:51.320192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 574/1144] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:51 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:35:51.326755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2c08d7-c3f5-4099-add9-d89359cf697b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:51.356271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:51.423864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2c08d7-c3f5-4099-add9-d89359cf697b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:51.457123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f9c804-1fb8-4292-8a1c-bfb88b365b6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:51.457734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f9c804-1fb8-4292-8a1c-bfb88b365b6d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:51.458022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f9c804-1fb8-4292-8a1c-bfb88b365b6d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:51.458297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f9c804-1fb8-4292-8a1c-bfb88b365b6d None admin] PATH_INFO: `/v3/projects/ce870b9aaaca4c6db85cb7b31c51cebb` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:51.459431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f9c804-1fb8-4292-8a1c-bfb88b365b6d None admin] RBAC: Authorizing `identity:get_project(project_id=ce870b9aaaca4c6db85cb7b31c51cebb)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:51.460876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f9c804-1fb8-4292-8a1c-bfb88b365b6d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:51.462514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 571/1145] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:35:51 2022] GET /identity/v3/projects/ce870b9aaaca4c6db85cb7b31c51cebb => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:51.468868 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd7f979-7d0f-4c6d-9bf3-1c628fa8ae0d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:51.495062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:51.566207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd7f979-7d0f-4c6d-9bf3-1c628fa8ae0d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:51.598459 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a184b0c4-a06d-4440-a091-b2e6f406c3a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:51.599064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a184b0c4-a06d-4440-a091-b2e6f406c3a8 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:51.599346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a184b0c4-a06d-4440-a091-b2e6f406c3a8 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:51.599622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a184b0c4-a06d-4440-a091-b2e6f406c3a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:51.600401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a184b0c4-a06d-4440-a091-b2e6f406c3a8 None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=ce870b9aaaca4c6db85cb7b31c51cebb, user.id=2cbbcf61163049e293aa90eaa3aa0ce2) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:51.600740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a184b0c4-a06d-4440-a091-b2e6f406c3a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:51.602169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a184b0c4-a06d-4440-a091-b2e6f406c3a8 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:51.607930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 575/1146] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:51 2022] GET /identity/v3/role_assignments?user.id=2cbbcf61163049e293aa90eaa3aa0ce2&scope.project.id=ce870b9aaaca4c6db85cb7b31c51cebb&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:53.578512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167fc8a-5766-4084-8000-e71284fc8306 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:53.579079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167fc8a-5766-4084-8000-e71284fc8306 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:53.579079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6167fc8a-5766-4084-8000-e71284fc8306 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:53.580562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 572/1147] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:53.585825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b29b741-fa7f-4a6b-b508-f96739588f71 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:53.586251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b29b741-fa7f-4a6b-b508-f96739588f71 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:53.586386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b29b741-fa7f-4a6b-b508-f96739588f71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:53.604780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4b29b741-fa7f-4a6b-b508-f96739588f71 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:53.605982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4b29b741-fa7f-4a6b-b508-f96739588f71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:53.671341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 576/1148] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:53 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:53.693694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6167fc8a-5766-4084-8000-e71284fc8306 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:53.788277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:53.792485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6167fc8a-5766-4084-8000-e71284fc8306 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:53.830707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7324a6-eac0-417e-9f82-4e8fb31bebbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:53.831278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7324a6-eac0-417e-9f82-4e8fb31bebbe None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:53.831569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7324a6-eac0-417e-9f82-4e8fb31bebbe None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:53.831853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7324a6-eac0-417e-9f82-4e8fb31bebbe None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:53.842290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7324a6-eac0-417e-9f82-4e8fb31bebbe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-ea7324a6-eac0-417e-9f82-4e8fb31bebbe None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:53.847171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:53.848252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:35:53.849302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 573/1149] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:35:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:53.854357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b29b741-fa7f-4a6b-b508-f96739588f71 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:53.875123 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:53.953301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b29b741-fa7f-4a6b-b508-f96739588f71 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:53.986391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af9bdff-20a8-4f00-b765-15d016ed891a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:53.986994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af9bdff-20a8-4f00-b765-15d016ed891a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:53.987280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af9bdff-20a8-4f00-b765-15d016ed891a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:53.987553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af9bdff-20a8-4f00-b765-15d016ed891a None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:53.988262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af9bdff-20a8-4f00-b765-15d016ed891a None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:53.988565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af9bdff-20a8-4f00-b765-15d016ed891a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:53.989955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af9bdff-20a8-4f00-b765-15d016ed891a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:53.998182 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 577/1150] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:35:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:54.003930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7324a6-eac0-417e-9f82-4e8fb31bebbe None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:54.101919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7324a6-eac0-417e-9f82-4e8fb31bebbe None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:54.134929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e361bb-e30b-4913-8d16-97491bce9378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:54.135562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e361bb-e30b-4913-8d16-97491bce9378 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:54.135851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e361bb-e30b-4913-8d16-97491bce9378 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:54.136134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e361bb-e30b-4913-8d16-97491bce9378 None admin] PATH_INFO: `/v3/users/2cbbcf61163049e293aa90eaa3aa0ce2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:54.137270 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e361bb-e30b-4913-8d16-97491bce9378 None admin] RBAC: Authorizing `identity:get_user(user_id=2cbbcf61163049e293aa90eaa3aa0ce2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:54.139322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e361bb-e30b-4913-8d16-97491bce9378 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:54.140957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 574/1151] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:54 2022] GET /identity/v3/users/2cbbcf61163049e293aa90eaa3aa0ce2 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:54.146838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af9bdff-20a8-4f00-b765-15d016ed891a None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:54.225881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:54.243865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af9bdff-20a8-4f00-b765-15d016ed891a None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:54.277877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef81bae-6615-483e-aef4-20add8723585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:54.278515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef81bae-6615-483e-aef4-20add8723585 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:54.278797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef81bae-6615-483e-aef4-20add8723585 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:54.279075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef81bae-6615-483e-aef4-20add8723585 None admin] PATH_INFO: `/v3/projects/ce870b9aaaca4c6db85cb7b31c51cebb` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:54.280566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef81bae-6615-483e-aef4-20add8723585 None admin] RBAC: Authorizing `identity:get_project(project_id=ce870b9aaaca4c6db85cb7b31c51cebb)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:54.282029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef81bae-6615-483e-aef4-20add8723585 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:54.283678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 578/1152] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:35:54 2022] GET /identity/v3/projects/ce870b9aaaca4c6db85cb7b31c51cebb => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:54.289753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e361bb-e30b-4913-8d16-97491bce9378 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:54.318111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:54.389081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e361bb-e30b-4913-8d16-97491bce9378 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:54.423882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ece96a8-0329-4875-b5ad-b519dbd53c93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:54.424543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece96a8-0329-4875-b5ad-b519dbd53c93 None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:54.424863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece96a8-0329-4875-b5ad-b519dbd53c93 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:54.425146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece96a8-0329-4875-b5ad-b519dbd53c93 None admin] PATH_INFO: `/v3/projects/ce870b9aaaca4c6db85cb7b31c51cebb/users/2cbbcf61163049e293aa90eaa3aa0ce2/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:54.427124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ece96a8-0329-4875-b5ad-b519dbd53c93 None admin] RBAC: Authorizing `identity:create_grant(project_id=ce870b9aaaca4c6db85cb7b31c51cebb, user_id=2cbbcf61163049e293aa90eaa3aa0ce2, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:54.428549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ece96a8-0329-4875-b5ad-b519dbd53c93 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:54.441550 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 575/1153] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:35:54 2022] PUT /identity/v3/projects/ce870b9aaaca4c6db85cb7b31c51cebb/users/2cbbcf61163049e293aa90eaa3aa0ce2/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:35:56.402951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed826e95-c244-4b1d-90c0-bdb123980250 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:56.403581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed826e95-c244-4b1d-90c0-bdb123980250 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:56.403581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed826e95-c244-4b1d-90c0-bdb123980250 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:56.405075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 579/1154] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:56.410428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7ab215-c38f-402c-a931-9f2adb609dae None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:56.410818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7ab215-c38f-402c-a931-9f2adb609dae None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:56.410995 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7ab215-c38f-402c-a931-9f2adb609dae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:56.429332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-8b7ab215-c38f-402c-a931-9f2adb609dae None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:56.430536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8b7ab215-c38f-402c-a931-9f2adb609dae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:56.495846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 576/1155] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:56 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:56.519855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed826e95-c244-4b1d-90c0-bdb123980250 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:56.617762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed826e95-c244-4b1d-90c0-bdb123980250 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:56.650294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6741e65a-dbd8-4176-b3cc-e93b08847fac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:56.650926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6741e65a-dbd8-4176-b3cc-e93b08847fac None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:56.651206 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6741e65a-dbd8-4176-b3cc-e93b08847fac None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:56.651479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6741e65a-dbd8-4176-b3cc-e93b08847fac None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:56.661567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6741e65a-dbd8-4176-b3cc-e93b08847fac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-6741e65a-dbd8-4176-b3cc-e93b08847fac None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:35:56.666505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:35:56.667571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:35:56.668625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 580/1156] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:35:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:35:56.674745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7ab215-c38f-402c-a931-9f2adb609dae None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:56.693717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:56.772979 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7ab215-c38f-402c-a931-9f2adb609dae None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:56.806058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccba8c4f-983e-4301-ab13-040e9e9ac153 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:56.806671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccba8c4f-983e-4301-ab13-040e9e9ac153 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:56.806960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccba8c4f-983e-4301-ab13-040e9e9ac153 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:56.807239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccba8c4f-983e-4301-ab13-040e9e9ac153 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:56.807958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccba8c4f-983e-4301-ab13-040e9e9ac153 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:56.808267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccba8c4f-983e-4301-ab13-040e9e9ac153 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:56.810287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccba8c4f-983e-4301-ab13-040e9e9ac153 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:56.818161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 577/1157] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:35:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:56.823895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6741e65a-dbd8-4176-b3cc-e93b08847fac None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:56.836365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:56.922521 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6741e65a-dbd8-4176-b3cc-e93b08847fac None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:56.955533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e99ad-361f-41af-b351-a4da2bdb982b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:56.956168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e99ad-361f-41af-b351-a4da2bdb982b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:56.956474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e99ad-361f-41af-b351-a4da2bdb982b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:56.956782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e99ad-361f-41af-b351-a4da2bdb982b None admin] PATH_INFO: `/v3/users/2cbbcf61163049e293aa90eaa3aa0ce2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:56.957954 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3e99ad-361f-41af-b351-a4da2bdb982b None admin] RBAC: Authorizing `identity:get_user(user_id=2cbbcf61163049e293aa90eaa3aa0ce2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:56.959384 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3e99ad-361f-41af-b351-a4da2bdb982b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:56.961019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 581/1158] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:35:56 2022] GET /identity/v3/users/2cbbcf61163049e293aa90eaa3aa0ce2 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:56.967278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccba8c4f-983e-4301-ab13-040e9e9ac153 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:57.046451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:57.065228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccba8c4f-983e-4301-ab13-040e9e9ac153 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:57.097953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9915eb41-fd76-41de-964c-c3992e057891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:57.098558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9915eb41-fd76-41de-964c-c3992e057891 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:57.098840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9915eb41-fd76-41de-964c-c3992e057891 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:57.099114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9915eb41-fd76-41de-964c-c3992e057891 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:57.163340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9915eb41-fd76-41de-964c-c3992e057891 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:57.164801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9915eb41-fd76-41de-964c-c3992e057891 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:57.240593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:57.259978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 578/1159] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:56 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:35:57.266499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e99ad-361f-41af-b351-a4da2bdb982b None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:57.343739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:57.364938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e99ad-361f-41af-b351-a4da2bdb982b None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:57.398790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7d4f37-54ad-47c2-a6f5-49091908139f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:57.399415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7d4f37-54ad-47c2-a6f5-49091908139f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:57.399697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7d4f37-54ad-47c2-a6f5-49091908139f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:57.399968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7d4f37-54ad-47c2-a6f5-49091908139f None admin] PATH_INFO: `/v3/projects/ce870b9aaaca4c6db85cb7b31c51cebb` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:57.401126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7d4f37-54ad-47c2-a6f5-49091908139f None admin] RBAC: Authorizing `identity:get_project(project_id=ce870b9aaaca4c6db85cb7b31c51cebb)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:57.402543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7d4f37-54ad-47c2-a6f5-49091908139f None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:57.404881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 582/1160] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:35:57 2022] GET /identity/v3/projects/ce870b9aaaca4c6db85cb7b31c51cebb => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:57.411123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9915eb41-fd76-41de-964c-c3992e057891 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:57.508674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9915eb41-fd76-41de-964c-c3992e057891 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:57.542352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18758799-4a49-4295-9a70-38d943f9e056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:57.542962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18758799-4a49-4295-9a70-38d943f9e056 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:57.543252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18758799-4a49-4295-9a70-38d943f9e056 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:57.543532 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18758799-4a49-4295-9a70-38d943f9e056 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:57.544517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18758799-4a49-4295-9a70-38d943f9e056 None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=ce870b9aaaca4c6db85cb7b31c51cebb, user.id=2cbbcf61163049e293aa90eaa3aa0ce2) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:35:57.544871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18758799-4a49-4295-9a70-38d943f9e056 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:57.546291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18758799-4a49-4295-9a70-38d943f9e056 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:57.552185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 579/1161] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:35:57 2022] GET /identity/v3/role_assignments?user.id=2cbbcf61163049e293aa90eaa3aa0ce2&scope.project.id=ce870b9aaaca4c6db85cb7b31c51cebb&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:35:59.552140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848ce28d-0ec1-46cc-94e1-879c5e3aac88 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:59.552717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848ce28d-0ec1-46cc-94e1-879c5e3aac88 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:59.552717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848ce28d-0ec1-46cc-94e1-879c5e3aac88 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:59.554087 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 583/1162] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:35:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:35:59.559301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000fabeb-e963-4a0a-a14e-27592b98937a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:59.560219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000fabeb-e963-4a0a-a14e-27592b98937a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:59.560633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000fabeb-e963-4a0a-a14e-27592b98937a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:59.579102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-000fabeb-e963-4a0a-a14e-27592b98937a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:35:59.580290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-000fabeb-e963-4a0a-a14e-27592b98937a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:35:59.618887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:59.645818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 580/1163] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:35:59 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:35:59.667861 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848ce28d-0ec1-46cc-94e1-879c5e3aac88 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:35:59.765606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848ce28d-0ec1-46cc-94e1-879c5e3aac88 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:35:59.799129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080bc2be-83b9-4dc7-aec9-71147f83e8e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:35:59.799732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080bc2be-83b9-4dc7-aec9-71147f83e8e9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:35:59.800015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080bc2be-83b9-4dc7-aec9-71147f83e8e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:35:59.800288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080bc2be-83b9-4dc7-aec9-71147f83e8e9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:35:59.815268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:35:59.865323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080bc2be-83b9-4dc7-aec9-71147f83e8e9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:35:59.866756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080bc2be-83b9-4dc7-aec9-71147f83e8e9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:35:59.960226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 584/1164] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:35:59 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:35:59.966739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000fabeb-e963-4a0a-a14e-27592b98937a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:00.063256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000fabeb-e963-4a0a-a14e-27592b98937a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:00.096152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27229d04-31d8-476a-9a9d-74f696a895c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:00.096760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27229d04-31d8-476a-9a9d-74f696a895c7 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:00.097050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27229d04-31d8-476a-9a9d-74f696a895c7 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:00.097331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27229d04-31d8-476a-9a9d-74f696a895c7 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:00.103831 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27229d04-31d8-476a-9a9d-74f696a895c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:00.105300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27229d04-31d8-476a-9a9d-74f696a895c7 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-27229d04-31d8-476a-9a9d-74f696a895c7 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:36:00.117191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:36:00.118371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:36:00.119364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 581/1165] 10.222.0.40 () {56 vars in 1117 bytes} [Fri Jul 29 04:35:59 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:36:00.124427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080bc2be-83b9-4dc7-aec9-71147f83e8e9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:00.169855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:00.221283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080bc2be-83b9-4dc7-aec9-71147f83e8e9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:00.254091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83767e94-ff59-4b83-8a2e-1694b3814f34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:00.254697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83767e94-ff59-4b83-8a2e-1694b3814f34 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:00.254982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83767e94-ff59-4b83-8a2e-1694b3814f34 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:00.255260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83767e94-ff59-4b83-8a2e-1694b3814f34 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:00.255947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83767e94-ff59-4b83-8a2e-1694b3814f34 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:36:00.256251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83767e94-ff59-4b83-8a2e-1694b3814f34 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:00.257643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83767e94-ff59-4b83-8a2e-1694b3814f34 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:00.264783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 585/1166] 10.222.0.40 () {56 vars in 1115 bytes} [Fri Jul 29 04:36:00 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:02.218601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a278e5d2-a712-4db2-98de-ff2d37577688 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:02.219130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a278e5d2-a712-4db2-98de-ff2d37577688 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:02.219130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a278e5d2-a712-4db2-98de-ff2d37577688 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:02.220508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 582/1167] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:36:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:36:02.225872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb5492-7313-439c-9b2f-3710b7f6436d None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:02.226291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb5492-7313-439c-9b2f-3710b7f6436d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:02.226443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb5492-7313-439c-9b2f-3710b7f6436d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:02.244952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-45cb5492-7313-439c-9b2f-3710b7f6436d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:36:02.246149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-45cb5492-7313-439c-9b2f-3710b7f6436d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:36:02.311412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 586/1168] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:36:02 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:36:02.334395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a278e5d2-a712-4db2-98de-ff2d37577688 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:02.369763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:02.434764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a278e5d2-a712-4db2-98de-ff2d37577688 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:02.466899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71ff8b2-94fa-4b1f-a02b-1827d911bb09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:02.467502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ff8b2-94fa-4b1f-a02b-1827d911bb09 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:02.467790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ff8b2-94fa-4b1f-a02b-1827d911bb09 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:02.468069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71ff8b2-94fa-4b1f-a02b-1827d911bb09 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:02.468864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71ff8b2-94fa-4b1f-a02b-1827d911bb09 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:02.470180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71ff8b2-94fa-4b1f-a02b-1827d911bb09 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:02.485958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 583/1169] 10.222.0.40 () {60 vars in 1138 bytes} [Fri Jul 29 04:36:02 2022] POST /identity/v3/domains => generated 267 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:36:04.485275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6428b7-6fce-4b13-81d7-8cd5100f7cc4 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:04.485806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6428b7-6fce-4b13-81d7-8cd5100f7cc4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:04.485898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6428b7-6fce-4b13-81d7-8cd5100f7cc4 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:04.487202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 587/1170] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:36:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:36:04.492317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b862d4a0-e0df-4f4d-8e30-e49a1b982102 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:04.492739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b862d4a0-e0df-4f4d-8e30-e49a1b982102 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:04.492901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b862d4a0-e0df-4f4d-8e30-e49a1b982102 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:04.510898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-b862d4a0-e0df-4f4d-8e30-e49a1b982102 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:36:04.512107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b862d4a0-e0df-4f4d-8e30-e49a1b982102 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:36:04.577815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 584/1171] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:36:04 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:36:04.600988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6428b7-6fce-4b13-81d7-8cd5100f7cc4 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:04.642640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:04.699810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6428b7-6fce-4b13-81d7-8cd5100f7cc4 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:04.732620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dc5da6-09ca-4ecc-8284-a71cceb0aa31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:04.733268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc5da6-09ca-4ecc-8284-a71cceb0aa31 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:04.733548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc5da6-09ca-4ecc-8284-a71cceb0aa31 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:04.733821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dc5da6-09ca-4ecc-8284-a71cceb0aa31 None admin] PATH_INFO: `/v3/domains/ad635a76cdec4696869ae3b32a62c511` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:04.743918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dc5da6-09ca-4ecc-8284-a71cceb0aa31 None admin] RBAC: Authorizing `identity:get_domain(domain_id=ad635a76cdec4696869ae3b32a62c511)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:04.745337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dc5da6-09ca-4ecc-8284-a71cceb0aa31 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:04.748147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 588/1172] 10.222.0.40 () {56 vars in 1183 bytes} [Fri Jul 29 04:36:04 2022] GET /identity/v3/domains/ad635a76cdec4696869ae3b32a62c511 => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:04.754243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b862d4a0-e0df-4f4d-8e30-e49a1b982102 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:04.778552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:04.851913 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b862d4a0-e0df-4f4d-8e30-e49a1b982102 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:04.884250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec77c55c-675f-44ce-9af6-f84c9464bf35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:04.884893 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec77c55c-675f-44ce-9af6-f84c9464bf35 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:04.885183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec77c55c-675f-44ce-9af6-f84c9464bf35 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:04.885462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec77c55c-675f-44ce-9af6-f84c9464bf35 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:04.886143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec77c55c-675f-44ce-9af6-f84c9464bf35 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:04.888294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec77c55c-675f-44ce-9af6-f84c9464bf35 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:04.913768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 585/1173] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:36:04 2022] POST /identity/v3/projects => generated 361 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:36:06.948712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9462d79-67c7-4dee-9581-043159e2676a None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:06.949222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9462d79-67c7-4dee-9581-043159e2676a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:06.949222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9462d79-67c7-4dee-9581-043159e2676a None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:06.951218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 589/1174] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:36:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:36:06.956315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ed12db-f5f6-4c47-9e38-80c7269b586c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:06.956711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ed12db-f5f6-4c47-9e38-80c7269b586c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:06.956909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ed12db-f5f6-4c47-9e38-80c7269b586c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:06.975730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-78ed12db-f5f6-4c47-9e38-80c7269b586c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:36:06.976946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-78ed12db-f5f6-4c47-9e38-80c7269b586c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:36:07.041146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 586/1175] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:36:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:36:07.066157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9462d79-67c7-4dee-9581-043159e2676a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:07.161770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:07.166089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9462d79-67c7-4dee-9581-043159e2676a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:07.201193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3dbf515-f450-4227-b015-a1948dfcd26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:07.201763 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dbf515-f450-4227-b015-a1948dfcd26a None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:07.202047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dbf515-f450-4227-b015-a1948dfcd26a None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:07.202322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3dbf515-f450-4227-b015-a1948dfcd26a None admin] PATH_INFO: `/v3/domains/ad635a76cdec4696869ae3b32a62c511` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:07.203470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3dbf515-f450-4227-b015-a1948dfcd26a None admin] RBAC: Authorizing `identity:get_domain(domain_id=ad635a76cdec4696869ae3b32a62c511)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:07.204917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3dbf515-f450-4227-b015-a1948dfcd26a None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:07.206484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 590/1176] 10.222.0.40 () {56 vars in 1183 bytes} [Fri Jul 29 04:36:07 2022] GET /identity/v3/domains/ad635a76cdec4696869ae3b32a62c511 => generated 267 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:07.212195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ed12db-f5f6-4c47-9e38-80c7269b586c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:07.220266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:07.312113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ed12db-f5f6-4c47-9e38-80c7269b586c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:07.346333 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75d84e5-46db-44b3-96d2-af009a3cd5fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:07.346949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75d84e5-46db-44b3-96d2-af009a3cd5fc None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:07.347241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75d84e5-46db-44b3-96d2-af009a3cd5fc None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:07.347521 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75d84e5-46db-44b3-96d2-af009a3cd5fc None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:07.348199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75d84e5-46db-44b3-96d2-af009a3cd5fc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:07.349664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f75d84e5-46db-44b3-96d2-af009a3cd5fc None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:07.371095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 587/1177] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:36:07 2022] POST /identity/v3/users => generated 311 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:36:09.415507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac418d0b-6851-448c-918f-a25e3b72f3ba None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:09.416138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac418d0b-6851-448c-918f-a25e3b72f3ba None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:09.416138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac418d0b-6851-448c-918f-a25e3b72f3ba None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:09.417444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 591/1178] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:36:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:36:09.423478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f91674-87e8-4c03-bf5c-4df9e5a33e09 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:09.423882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f91674-87e8-4c03-bf5c-4df9e5a33e09 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:09.424036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f91674-87e8-4c03-bf5c-4df9e5a33e09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:09.442526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-53f91674-87e8-4c03-bf5c-4df9e5a33e09 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:36:09.443715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-53f91674-87e8-4c03-bf5c-4df9e5a33e09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:36:09.498214 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:09.508651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 588/1179] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:36:09 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:36:09.531353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac418d0b-6851-448c-918f-a25e3b72f3ba None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:09.631287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac418d0b-6851-448c-918f-a25e3b72f3ba None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:09.663932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2714c9-f607-4e2f-9a69-507bcb942c5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:09.664582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2714c9-f607-4e2f-9a69-507bcb942c5f None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:09.664902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2714c9-f607-4e2f-9a69-507bcb942c5f None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:09.665184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2714c9-f607-4e2f-9a69-507bcb942c5f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:09.676121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af2714c9-f607-4e2f-9a69-507bcb942c5f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-af2714c9-f607-4e2f-9a69-507bcb942c5f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:36:09.680968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:36:09.682160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:36:09.683275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 592/1180] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:36:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:36:09.689007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f91674-87e8-4c03-bf5c-4df9e5a33e09 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:09.783940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f91674-87e8-4c03-bf5c-4df9e5a33e09 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:09.816652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bf94b4-eb82-4a28-a7b2-23aa401e3b74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:09.817264 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf94b4-eb82-4a28-a7b2-23aa401e3b74 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:09.817549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf94b4-eb82-4a28-a7b2-23aa401e3b74 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:09.817826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bf94b4-eb82-4a28-a7b2-23aa401e3b74 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:09.818536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bf94b4-eb82-4a28-a7b2-23aa401e3b74 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:36:09.818845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bf94b4-eb82-4a28-a7b2-23aa401e3b74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:09.820209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bf94b4-eb82-4a28-a7b2-23aa401e3b74 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:09.828039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 589/1181] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:36:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:36:09.833920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2714c9-f607-4e2f-9a69-507bcb942c5f None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:09.863409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:09.939659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2714c9-f607-4e2f-9a69-507bcb942c5f None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:09.972623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4204cb-e583-46a9-9b7e-376e17d5faaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:09.973267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4204cb-e583-46a9-9b7e-376e17d5faaf None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:09.973547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4204cb-e583-46a9-9b7e-376e17d5faaf None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:09.973822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4204cb-e583-46a9-9b7e-376e17d5faaf None admin] PATH_INFO: `/v3/users/d99097dd02fa48378b956e87d05fc339` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:09.987207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4204cb-e583-46a9-9b7e-376e17d5faaf None admin] RBAC: Authorizing `identity:get_user(user_id=d99097dd02fa48378b956e87d05fc339)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:09.988671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4204cb-e583-46a9-9b7e-376e17d5faaf None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:09.990275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 593/1182] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:36:09 2022] GET /identity/v3/users/d99097dd02fa48378b956e87d05fc339 => generated 311 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:09.996202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bf94b4-eb82-4a28-a7b2-23aa401e3b74 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:10.051508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:10.096359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bf94b4-eb82-4a28-a7b2-23aa401e3b74 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:10.128719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94a946f-70c9-496c-826e-d79c13c56afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:10.129339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94a946f-70c9-496c-826e-d79c13c56afc None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:10.129628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94a946f-70c9-496c-826e-d79c13c56afc None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:10.129907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94a946f-70c9-496c-826e-d79c13c56afc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:10.195619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94a946f-70c9-496c-826e-d79c13c56afc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:10.197044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94a946f-70c9-496c-826e-d79c13c56afc None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:10.249998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:10.294892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 590/1183] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:36:09 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:36:10.301245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4204cb-e583-46a9-9b7e-376e17d5faaf None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:10.384081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:10.399670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4204cb-e583-46a9-9b7e-376e17d5faaf None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:10.432555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613696de-caa9-4f66-9737-576dd2c6ce64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:10.433211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613696de-caa9-4f66-9737-576dd2c6ce64 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:10.433496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613696de-caa9-4f66-9737-576dd2c6ce64 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:10.434262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613696de-caa9-4f66-9737-576dd2c6ce64 None admin] PATH_INFO: `/v3/projects/8404c6f7229543d8bce9704c4f8285d7` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:10.435407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613696de-caa9-4f66-9737-576dd2c6ce64 None admin] RBAC: Authorizing `identity:get_project(project_id=8404c6f7229543d8bce9704c4f8285d7)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:10.436859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613696de-caa9-4f66-9737-576dd2c6ce64 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:10.438495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 594/1184] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:36:10 2022] GET /identity/v3/projects/8404c6f7229543d8bce9704c4f8285d7 => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:10.444569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94a946f-70c9-496c-826e-d79c13c56afc None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:10.541184 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94a946f-70c9-496c-826e-d79c13c56afc None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:10.574737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f133b9eb-28bc-4596-8e1e-a01b1f9d72fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:10.575349 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f133b9eb-28bc-4596-8e1e-a01b1f9d72fe None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:10.575639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f133b9eb-28bc-4596-8e1e-a01b1f9d72fe None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:10.575919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f133b9eb-28bc-4596-8e1e-a01b1f9d72fe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:10.576739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f133b9eb-28bc-4596-8e1e-a01b1f9d72fe None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=8404c6f7229543d8bce9704c4f8285d7, user.id=d99097dd02fa48378b956e87d05fc339) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:36:10.577059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f133b9eb-28bc-4596-8e1e-a01b1f9d72fe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:10.578509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f133b9eb-28bc-4596-8e1e-a01b1f9d72fe None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:10.583796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 591/1185] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:36:10 2022] GET /identity/v3/role_assignments?user.id=d99097dd02fa48378b956e87d05fc339&scope.project.id=8404c6f7229543d8bce9704c4f8285d7&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:12.559041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c504c21-dfe1-4243-9921-3d12edc484be None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:12.559598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c504c21-dfe1-4243-9921-3d12edc484be None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:12.559598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c504c21-dfe1-4243-9921-3d12edc484be None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:12.560976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 595/1186] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:36:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:36:12.566063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b8d481-531a-4cb8-84e6-15504d0499a3 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:12.566455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b8d481-531a-4cb8-84e6-15504d0499a3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:12.566635 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b8d481-531a-4cb8-84e6-15504d0499a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:12.585789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-e8b8d481-531a-4cb8-84e6-15504d0499a3 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:36:12.586988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e8b8d481-531a-4cb8-84e6-15504d0499a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:36:12.600749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:12.654337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 592/1187] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:36:12 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:36:12.677087 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c504c21-dfe1-4243-9921-3d12edc484be None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:12.776510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c504c21-dfe1-4243-9921-3d12edc484be None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:12.811132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7988d0bb-46d9-4f25-ae34-3556bbf93fb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:12.811701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988d0bb-46d9-4f25-ae34-3556bbf93fb8 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:12.811988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988d0bb-46d9-4f25-ae34-3556bbf93fb8 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:12.812269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988d0bb-46d9-4f25-ae34-3556bbf93fb8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:12.822125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7988d0bb-46d9-4f25-ae34-3556bbf93fb8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-7988d0bb-46d9-4f25-ae34-3556bbf93fb8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:36:12.827164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:36:12.828257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:36:12.829311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 596/1188] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:36:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:36:12.834478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b8d481-531a-4cb8-84e6-15504d0499a3 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:12.931079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b8d481-531a-4cb8-84e6-15504d0499a3 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:12.963378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e118da70-3fe5-4413-a39d-6a65182ceafd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:12.963990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118da70-3fe5-4413-a39d-6a65182ceafd None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:12.964277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118da70-3fe5-4413-a39d-6a65182ceafd None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:12.964557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e118da70-3fe5-4413-a39d-6a65182ceafd None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:12.965860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e118da70-3fe5-4413-a39d-6a65182ceafd None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:36:12.966172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e118da70-3fe5-4413-a39d-6a65182ceafd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:12.967541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e118da70-3fe5-4413-a39d-6a65182ceafd None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:12.975362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 593/1189] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:36:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:12.981554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7988d0bb-46d9-4f25-ae34-3556bbf93fb8 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:12.997767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:13.080498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7988d0bb-46d9-4f25-ae34-3556bbf93fb8 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:13.113182 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52e0cba-dc35-48fe-9dad-bb4749df3613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:13.113818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52e0cba-dc35-48fe-9dad-bb4749df3613 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:13.114103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52e0cba-dc35-48fe-9dad-bb4749df3613 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:13.114381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52e0cba-dc35-48fe-9dad-bb4749df3613 None admin] PATH_INFO: `/v3/users/d99097dd02fa48378b956e87d05fc339` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:13.115535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52e0cba-dc35-48fe-9dad-bb4749df3613 None admin] RBAC: Authorizing `identity:get_user(user_id=d99097dd02fa48378b956e87d05fc339)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:13.117010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d52e0cba-dc35-48fe-9dad-bb4749df3613 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:13.118591 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 597/1190] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:36:12 2022] GET /identity/v3/users/d99097dd02fa48378b956e87d05fc339 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:13.124424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e118da70-3fe5-4413-a39d-6a65182ceafd None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:13.137596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:13.224354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e118da70-3fe5-4413-a39d-6a65182ceafd None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:13.256646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6666ddf9-006a-4291-a768-0a03dd9f426a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:13.257286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666ddf9-006a-4291-a768-0a03dd9f426a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:13.257571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666ddf9-006a-4291-a768-0a03dd9f426a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:13.257852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6666ddf9-006a-4291-a768-0a03dd9f426a None admin] PATH_INFO: `/v3/projects/8404c6f7229543d8bce9704c4f8285d7` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:13.258998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6666ddf9-006a-4291-a768-0a03dd9f426a None admin] RBAC: Authorizing `identity:get_project(project_id=8404c6f7229543d8bce9704c4f8285d7)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:13.260420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6666ddf9-006a-4291-a768-0a03dd9f426a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:13.262668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 594/1191] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:36:13 2022] GET /identity/v3/projects/8404c6f7229543d8bce9704c4f8285d7 => generated 361 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:13.268571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52e0cba-dc35-48fe-9dad-bb4749df3613 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:13.354691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:13.372895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52e0cba-dc35-48fe-9dad-bb4749df3613 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:13.406890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ac672a-beab-46e4-b5e0-a69d1a540aa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:13.407553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ac672a-beab-46e4-b5e0-a69d1a540aa6 None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:13.407834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ac672a-beab-46e4-b5e0-a69d1a540aa6 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:13.408112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ac672a-beab-46e4-b5e0-a69d1a540aa6 None admin] PATH_INFO: `/v3/projects/8404c6f7229543d8bce9704c4f8285d7/users/d99097dd02fa48378b956e87d05fc339/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:13.410068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ac672a-beab-46e4-b5e0-a69d1a540aa6 None admin] RBAC: Authorizing `identity:create_grant(project_id=8404c6f7229543d8bce9704c4f8285d7, user_id=d99097dd02fa48378b956e87d05fc339, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:13.411489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ac672a-beab-46e4-b5e0-a69d1a540aa6 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:13.423904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 598/1192] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:36:13 2022] PUT /identity/v3/projects/8404c6f7229543d8bce9704c4f8285d7/users/d99097dd02fa48378b956e87d05fc339/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:36:15.388685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36e890-b793-4536-905f-0191a8cef59b None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:15.389242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36e890-b793-4536-905f-0191a8cef59b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:15.389242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36e890-b793-4536-905f-0191a8cef59b None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:15.390588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 595/1193] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:36:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:36:15.395766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e5dba3-adfa-47dd-80cb-c5c553e36b29 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:15.396201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e5dba3-adfa-47dd-80cb-c5c553e36b29 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:15.396334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e5dba3-adfa-47dd-80cb-c5c553e36b29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:15.415301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-d6e5dba3-adfa-47dd-80cb-c5c553e36b29 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:36:15.416508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d6e5dba3-adfa-47dd-80cb-c5c553e36b29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:36:15.482436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 599/1194] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:36:15 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:36:15.504836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e36e890-b793-4536-905f-0191a8cef59b None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:15.583100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:15.605760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e36e890-b793-4536-905f-0191a8cef59b None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:15.638803 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2055c05-2111-4e8b-9cc5-dd1b7b4062cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:15.639382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2055c05-2111-4e8b-9cc5-dd1b7b4062cc None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:15.639669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2055c05-2111-4e8b-9cc5-dd1b7b4062cc None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:15.639952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2055c05-2111-4e8b-9cc5-dd1b7b4062cc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:15.650387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2055c05-2111-4e8b-9cc5-dd1b7b4062cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-d2055c05-2111-4e8b-9cc5-dd1b7b4062cc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:36:15.656045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:36:15.657149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:36:15.658191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 596/1195] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:36:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:36:15.663340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e5dba3-adfa-47dd-80cb-c5c553e36b29 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:15.731550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:15.760398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e5dba3-adfa-47dd-80cb-c5c553e36b29 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:15.794468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7420355-fe2e-42d9-bf6e-5e26677ece86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:15.795068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7420355-fe2e-42d9-bf6e-5e26677ece86 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:15.795352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7420355-fe2e-42d9-bf6e-5e26677ece86 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:15.795626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7420355-fe2e-42d9-bf6e-5e26677ece86 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:15.796342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7420355-fe2e-42d9-bf6e-5e26677ece86 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:36:15.796669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7420355-fe2e-42d9-bf6e-5e26677ece86 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:15.798300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7420355-fe2e-42d9-bf6e-5e26677ece86 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:15.805994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 600/1196] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:36:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:15.811990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2055c05-2111-4e8b-9cc5-dd1b7b4062cc None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:15.908265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2055c05-2111-4e8b-9cc5-dd1b7b4062cc None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:15.940434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a00a9-3960-49d6-8dea-078ba088d75b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:15.941086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a00a9-3960-49d6-8dea-078ba088d75b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:15.941370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a00a9-3960-49d6-8dea-078ba088d75b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:15.941644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a00a9-3960-49d6-8dea-078ba088d75b None admin] PATH_INFO: `/v3/users/d99097dd02fa48378b956e87d05fc339` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:15.942785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a00a9-3960-49d6-8dea-078ba088d75b None admin] RBAC: Authorizing `identity:get_user(user_id=d99097dd02fa48378b956e87d05fc339)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:15.944188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a00a9-3960-49d6-8dea-078ba088d75b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:15.945781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 597/1197] 10.222.0.40 () {56 vars in 1177 bytes} [Fri Jul 29 04:36:15 2022] GET /identity/v3/users/d99097dd02fa48378b956e87d05fc339 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:15.951666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7420355-fe2e-42d9-bf6e-5e26677ece86 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:16.049305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7420355-fe2e-42d9-bf6e-5e26677ece86 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:16.081882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a901fddb-a7eb-4794-9cfb-8878c8db9058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:16.082491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a901fddb-a7eb-4794-9cfb-8878c8db9058 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:16.082783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a901fddb-a7eb-4794-9cfb-8878c8db9058 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:16.083063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a901fddb-a7eb-4794-9cfb-8878c8db9058 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:16.085618 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:16.149096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a901fddb-a7eb-4794-9cfb-8878c8db9058 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:16.150544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a901fddb-a7eb-4794-9cfb-8878c8db9058 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:16.247791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 601/1198] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:36:15 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:36:16.253784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a00a9-3960-49d6-8dea-078ba088d75b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:16.263307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:16.355005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a00a9-3960-49d6-8dea-078ba088d75b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:16.388440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e80596-14e7-47cb-ac7e-d56a7baff918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:16.389107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e80596-14e7-47cb-ac7e-d56a7baff918 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:16.389649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e80596-14e7-47cb-ac7e-d56a7baff918 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:16.389649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e80596-14e7-47cb-ac7e-d56a7baff918 None admin] PATH_INFO: `/v3/projects/8404c6f7229543d8bce9704c4f8285d7` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:16.390783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e80596-14e7-47cb-ac7e-d56a7baff918 None admin] RBAC: Authorizing `identity:get_project(project_id=8404c6f7229543d8bce9704c4f8285d7)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:16.392205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e80596-14e7-47cb-ac7e-d56a7baff918 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:16.393876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 598/1199] 10.222.0.40 () {56 vars in 1186 bytes} [Fri Jul 29 04:36:16 2022] GET /identity/v3/projects/8404c6f7229543d8bce9704c4f8285d7 => generated 361 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:16.400068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a901fddb-a7eb-4794-9cfb-8878c8db9058 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:36:16.433052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:16.498628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a901fddb-a7eb-4794-9cfb-8878c8db9058 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:36:16.531542 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e149b61a-f5df-4a14-ae31-bca851634406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:36:16.532149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e149b61a-f5df-4a14-ae31-bca851634406 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:36:16.532435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e149b61a-f5df-4a14-ae31-bca851634406 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:36:16.532746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e149b61a-f5df-4a14-ae31-bca851634406 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:36:16.533533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e149b61a-f5df-4a14-ae31-bca851634406 None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=8404c6f7229543d8bce9704c4f8285d7, user.id=d99097dd02fa48378b956e87d05fc339) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:36:16.533847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e149b61a-f5df-4a14-ae31-bca851634406 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:36:16.535290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e149b61a-f5df-4a14-ae31-bca851634406 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:36:16.540965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 602/1200] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:36:16 2022] GET /identity/v3/role_assignments?user.id=d99097dd02fa48378b956e87d05fc339&scope.project.id=8404c6f7229543d8bce9704c4f8285d7&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:36:26.384207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=63 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:36:26.538355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=51 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:11.426235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3303ce39-4c25-4121-ae8b-12706b7d8e27 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:11.426912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3303ce39-4c25-4121-ae8b-12706b7d8e27 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:11.426912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3303ce39-4c25-4121-ae8b-12706b7d8e27 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:11.428331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 599/1201] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:37:11.433732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8953e5a-6696-453f-bc4e-48b73e11c857 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:11.434230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8953e5a-6696-453f-bc4e-48b73e11c857 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:11.434353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8953e5a-6696-453f-bc4e-48b73e11c857 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:11.453180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c8953e5a-6696-453f-bc4e-48b73e11c857 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:11.454589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c8953e5a-6696-453f-bc4e-48b73e11c857 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:11.541841 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 603/1202] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:11 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:37:11.569246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3303ce39-4c25-4121-ae8b-12706b7d8e27 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:11.719488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3303ce39-4c25-4121-ae8b-12706b7d8e27 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:11.769742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7eaff-49a6-4cae-8cfa-6369e11b63cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:11.770362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7eaff-49a6-4cae-8cfa-6369e11b63cb None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:11.770652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7eaff-49a6-4cae-8cfa-6369e11b63cb None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:11.770935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7eaff-49a6-4cae-8cfa-6369e11b63cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:11.785656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e7eaff-49a6-4cae-8cfa-6369e11b63cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:11.787901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e7eaff-49a6-4cae-8cfa-6369e11b63cb None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-a2e7eaff-49a6-4cae-8cfa-6369e11b63cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:11.805673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:11.806869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:11.807836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:11.807836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:11.807836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:11.807836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:11.808755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 600/1203] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:11.815400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8953e5a-6696-453f-bc4e-48b73e11c857 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:11.953301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8953e5a-6696-453f-bc4e-48b73e11c857 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:11.993708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:12.002290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bce4be-8f9b-4eb3-b72d-c73061deb1b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:12.003099 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bce4be-8f9b-4eb3-b72d-c73061deb1b7 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:12.003458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bce4be-8f9b-4eb3-b72d-c73061deb1b7 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:12.003830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bce4be-8f9b-4eb3-b72d-c73061deb1b7 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:12.004577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bce4be-8f9b-4eb3-b72d-c73061deb1b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:12.004980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bce4be-8f9b-4eb3-b72d-c73061deb1b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:12.006669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bce4be-8f9b-4eb3-b72d-c73061deb1b7 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:12.021243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 604/1204] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:37:12.030133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7eaff-49a6-4cae-8cfa-6369e11b63cb None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:12.099091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:12.176393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7eaff-49a6-4cae-8cfa-6369e11b63cb None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:12.227929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3430a931-4d50-42de-804d-8d0619edfc84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:12.228814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3430a931-4d50-42de-804d-8d0619edfc84 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:12.229096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3430a931-4d50-42de-804d-8d0619edfc84 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:12.229475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3430a931-4d50-42de-804d-8d0619edfc84 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:12.230350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3430a931-4d50-42de-804d-8d0619edfc84 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:12.231981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3430a931-4d50-42de-804d-8d0619edfc84 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:12.270589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 601/1205] 10.222.0.40 () {60 vars in 1132 bytes} [Fri Jul 29 04:37:12 2022] POST /identity/v3/users => generated 251 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:37:14.475364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099f17b1-29d1-4557-909b-80464ce8c34c None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:14.475949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099f17b1-29d1-4557-909b-80464ce8c34c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:14.475949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099f17b1-29d1-4557-909b-80464ce8c34c None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:14.478547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 605/1206] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:37:14.484034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603c158b-84be-492c-9bf9-39d3c7cfc273 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:14.484496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603c158b-84be-492c-9bf9-39d3c7cfc273 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:14.484589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603c158b-84be-492c-9bf9-39d3c7cfc273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:14.506863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-603c158b-84be-492c-9bf9-39d3c7cfc273 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:14.508080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-603c158b-84be-492c-9bf9-39d3c7cfc273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:14.582368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 602/1207] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:14 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:37:14.607387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099f17b1-29d1-4557-909b-80464ce8c34c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:14.730639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099f17b1-29d1-4557-909b-80464ce8c34c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:14.769692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f300f718-45db-4fd7-9998-306ae462b7e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:14.770446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f300f718-45db-4fd7-9998-306ae462b7e4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:14.770735 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f300f718-45db-4fd7-9998-306ae462b7e4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:14.771116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f300f718-45db-4fd7-9998-306ae462b7e4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:14.782306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f300f718-45db-4fd7-9998-306ae462b7e4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-f300f718-45db-4fd7-9998-306ae462b7e4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:37:14.787613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:14.788627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:14.789466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 606/1208] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:37:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:14.797612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603c158b-84be-492c-9bf9-39d3c7cfc273 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:14.847829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:14.925820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603c158b-84be-492c-9bf9-39d3c7cfc273 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:14.968148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727f7c17-7095-4208-ae69-baf00f034481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:14.968813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727f7c17-7095-4208-ae69-baf00f034481 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:14.969096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727f7c17-7095-4208-ae69-baf00f034481 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:14.969448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727f7c17-7095-4208-ae69-baf00f034481 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:14.970174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727f7c17-7095-4208-ae69-baf00f034481 None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:14.970481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727f7c17-7095-4208-ae69-baf00f034481 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:14.971858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727f7c17-7095-4208-ae69-baf00f034481 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:14.982014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 603/1209] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:37:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:14.988061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f300f718-45db-4fd7-9998-306ae462b7e4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:15.060422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:15.121248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f300f718-45db-4fd7-9998-306ae462b7e4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:15.165512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fa4cd0-eddf-496e-ab0a-4c0353ee1152 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:15.166288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fa4cd0-eddf-496e-ab0a-4c0353ee1152 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:15.166573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fa4cd0-eddf-496e-ab0a-4c0353ee1152 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:15.166870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fa4cd0-eddf-496e-ab0a-4c0353ee1152 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:15.178124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fa4cd0-eddf-496e-ab0a-4c0353ee1152 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:15.179704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fa4cd0-eddf-496e-ab0a-4c0353ee1152 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-d5fa4cd0-eddf-496e-ab0a-4c0353ee1152 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:15.197674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:15.198678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:15.199562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:15.199562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:15.199562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:15.199562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:15.203702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 607/1210] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:15.204573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727f7c17-7095-4208-ae69-baf00f034481 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:15.312277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:15.315398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727f7c17-7095-4208-ae69-baf00f034481 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:15.353309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d30bbe-9979-4267-936c-57d0758405a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:15.354089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d30bbe-9979-4267-936c-57d0758405a2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:15.354381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d30bbe-9979-4267-936c-57d0758405a2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:15.354663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d30bbe-9979-4267-936c-57d0758405a2 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:15.355365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d30bbe-9979-4267-936c-57d0758405a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:15.355681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d30bbe-9979-4267-936c-57d0758405a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:15.357089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d30bbe-9979-4267-936c-57d0758405a2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:15.370753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 604/1211] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:15.376482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fa4cd0-eddf-496e-ab0a-4c0353ee1152 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:15.515773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fa4cd0-eddf-496e-ab0a-4c0353ee1152 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:15.517398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:15.561968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5394b2-f4e6-4876-ae9a-98213e5ffbdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:15.561968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5394b2-f4e6-4876-ae9a-98213e5ffbdd None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:15.561968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5394b2-f4e6-4876-ae9a-98213e5ffbdd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:15.561968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a5394b2-f4e6-4876-ae9a-98213e5ffbdd None admin] PATH_INFO: `/v3/users/placement` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:15.573202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5394b2-f4e6-4876-ae9a-98213e5ffbdd None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:15.575107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a5394b2-f4e6-4876-ae9a-98213e5ffbdd None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-6a5394b2-f4e6-4876-ae9a-98213e5ffbdd None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:37:15.591603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:15.593244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:15.594576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 608/1212] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:15 2022] GET /identity/v3/users/placement => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:15.598856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d30bbe-9979-4267-936c-57d0758405a2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:15.729220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d30bbe-9979-4267-936c-57d0758405a2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:15.767552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8848f6-65cb-4a2a-9a6c-4346ba1b928a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:15.768344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8848f6-65cb-4a2a-9a6c-4346ba1b928a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:15.768680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8848f6-65cb-4a2a-9a6c-4346ba1b928a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:15.769128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8848f6-65cb-4a2a-9a6c-4346ba1b928a None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:15.769979 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8848f6-65cb-4a2a-9a6c-4346ba1b928a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:15.770469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8848f6-65cb-4a2a-9a6c-4346ba1b928a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:15.772162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8848f6-65cb-4a2a-9a6c-4346ba1b928a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:15.795712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 605/1213] 10.222.0.40 () {56 vars in 1143 bytes} [Fri Jul 29 04:37:15 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:15.802269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5394b2-f4e6-4876-ae9a-98213e5ffbdd None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:15.932760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a5394b2-f4e6-4876-ae9a-98213e5ffbdd None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:15.971340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366de481-c8b9-453a-a788-3b4e2585fc10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:15.972720 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366de481-c8b9-453a-a788-3b4e2585fc10 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:15.973214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366de481-c8b9-453a-a788-3b4e2585fc10 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:15.973564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366de481-c8b9-453a-a788-3b4e2585fc10 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:16.014744 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:16.079522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366de481-c8b9-453a-a788-3b4e2585fc10 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:16.082400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366de481-c8b9-453a-a788-3b4e2585fc10 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:16.252014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 609/1214] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:37:15 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 29 04:37:16.257641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8848f6-65cb-4a2a-9a6c-4346ba1b928a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:16.305918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:16.409200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8848f6-65cb-4a2a-9a6c-4346ba1b928a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:16.447899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669c9a20-1927-48df-8208-876c9da0da7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:16.448776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669c9a20-1927-48df-8208-876c9da0da7a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:16.449160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669c9a20-1927-48df-8208-876c9da0da7a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:16.449538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669c9a20-1927-48df-8208-876c9da0da7a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:16.461118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669c9a20-1927-48df-8208-876c9da0da7a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:16.463453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669c9a20-1927-48df-8208-876c9da0da7a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-669c9a20-1927-48df-8208-876c9da0da7a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:16.479672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:16.481015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:16.482179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:16.482179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:16.482179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:16.482179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:16.483548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 606/1215] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:16.489205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366de481-c8b9-453a-a788-3b4e2585fc10 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:16.560762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:16.603036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366de481-c8b9-453a-a788-3b4e2585fc10 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:16.641983 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c1ca51-98db-41a5-b850-78c39588003d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:16.642628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c1ca51-98db-41a5-b850-78c39588003d None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:16.642916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c1ca51-98db-41a5-b850-78c39588003d None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:16.643192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c1ca51-98db-41a5-b850-78c39588003d None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:16.643964 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c1ca51-98db-41a5-b850-78c39588003d None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:16.644269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c1ca51-98db-41a5-b850-78c39588003d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:16.645768 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c1ca51-98db-41a5-b850-78c39588003d None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:16.660002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 610/1216] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:16.665242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669c9a20-1927-48df-8208-876c9da0da7a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:16.747289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:16.790666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669c9a20-1927-48df-8208-876c9da0da7a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:16.832690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af71bc63-1feb-4b2a-93c0-8ced6e21a22e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:16.833240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af71bc63-1feb-4b2a-93c0-8ced6e21a22e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:16.833526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af71bc63-1feb-4b2a-93c0-8ced6e21a22e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:16.833805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af71bc63-1feb-4b2a-93c0-8ced6e21a22e None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:16.841075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af71bc63-1feb-4b2a-93c0-8ced6e21a22e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:16.842570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af71bc63-1feb-4b2a-93c0-8ced6e21a22e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-af71bc63-1feb-4b2a-93c0-8ced6e21a22e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:37:16.857938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:16.859009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:16.859905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:37:16.859905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:37:16.859905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:37:16.859905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:37:16.859905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:37:16.859905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:37:16.859905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:16.859905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:16.859905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 607/1217] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:37:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:37:16.864874 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c1ca51-98db-41a5-b850-78c39588003d None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:16.984247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c1ca51-98db-41a5-b850-78c39588003d None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:17.010305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:17.028029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97d7a40-bb61-49a9-966c-6f1afbdc5f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:17.028739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d7a40-bb61-49a9-966c-6f1afbdc5f18 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:17.029049 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d7a40-bb61-49a9-966c-6f1afbdc5f18 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:17.029335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d7a40-bb61-49a9-966c-6f1afbdc5f18 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:17.030349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97d7a40-bb61-49a9-966c-6f1afbdc5f18 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:17.030732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97d7a40-bb61-49a9-966c-6f1afbdc5f18 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:17.032735 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97d7a40-bb61-49a9-966c-6f1afbdc5f18 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:17.056218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 611/1218] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:37:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:17.056426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af71bc63-1feb-4b2a-93c0-8ced6e21a22e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:17.188231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af71bc63-1feb-4b2a-93c0-8ced6e21a22e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:17.231675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:17.239112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1c80e1-4aa2-4914-9cda-0edc2d7c0390 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:17.239952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c80e1-4aa2-4914-9cda-0edc2d7c0390 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:17.240398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c80e1-4aa2-4914-9cda-0edc2d7c0390 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:17.240801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c80e1-4aa2-4914-9cda-0edc2d7c0390 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:17.241697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1c80e1-4aa2-4914-9cda-0edc2d7c0390 None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=e3da5e428049419a9b35494655f52484) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:17.242125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1c80e1-4aa2-4914-9cda-0edc2d7c0390 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:17.243693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1c80e1-4aa2-4914-9cda-0edc2d7c0390 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:17.250851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 608/1219] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:37:17 2022] GET /identity/v3/role_assignments?user.id=e3da5e428049419a9b35494655f52484&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 259 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:19.393616 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d867ece-cd66-404f-a56e-6c4884687ec5 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:19.394169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d867ece-cd66-404f-a56e-6c4884687ec5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:19.394259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d867ece-cd66-404f-a56e-6c4884687ec5 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:19.395572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 612/1220] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:37:19.401420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f302c0a6-6cd9-4451-b4d4-3922ddb28848 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:19.401843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f302c0a6-6cd9-4451-b4d4-3922ddb28848 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:19.402019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f302c0a6-6cd9-4451-b4d4-3922ddb28848 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:19.421166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f302c0a6-6cd9-4451-b4d4-3922ddb28848 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:19.422376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f302c0a6-6cd9-4451-b4d4-3922ddb28848 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:19.497391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 609/1221] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:19 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:37:19.521929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d867ece-cd66-404f-a56e-6c4884687ec5 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:19.629266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d867ece-cd66-404f-a56e-6c4884687ec5 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:19.671282 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e5126e-c4e5-41e8-b5c4-b65aff1f2696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:19.671938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e5126e-c4e5-41e8-b5c4-b65aff1f2696 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:19.672226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e5126e-c4e5-41e8-b5c4-b65aff1f2696 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:19.672507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e5126e-c4e5-41e8-b5c4-b65aff1f2696 None admin] PATH_INFO: `/v3/roles/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:19.683196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e5126e-c4e5-41e8-b5c4-b65aff1f2696 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-35e5126e-c4e5-41e8-b5c4-b65aff1f2696 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:37:19.688559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:19.689679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:19.690960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 613/1222] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:37:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:19.697202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f302c0a6-6cd9-4451-b4d4-3922ddb28848 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:19.809313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f302c0a6-6cd9-4451-b4d4-3922ddb28848 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:19.833611 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:19.846967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dfd044-c548-4a5d-82d7-0bbfc40fbfc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:19.847691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dfd044-c548-4a5d-82d7-0bbfc40fbfc8 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:19.848011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dfd044-c548-4a5d-82d7-0bbfc40fbfc8 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:19.848329 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dfd044-c548-4a5d-82d7-0bbfc40fbfc8 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:19.849304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dfd044-c548-4a5d-82d7-0bbfc40fbfc8 None admin] RBAC: Adding query filter params (name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:19.849659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dfd044-c548-4a5d-82d7-0bbfc40fbfc8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:19.851387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9dfd044-c548-4a5d-82d7-0bbfc40fbfc8 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:19.862487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 610/1223] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:37:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:19.868465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e5126e-c4e5-41e8-b5c4-b65aff1f2696 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:19.913220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:19.977576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e5126e-c4e5-41e8-b5c4-b65aff1f2696 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:20.011005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b6b866-2901-4da3-b6ce-3276bf1bcfc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:20.011647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b6b866-2901-4da3-b6ce-3276bf1bcfc6 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:20.011930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b6b866-2901-4da3-b6ce-3276bf1bcfc6 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:20.012204 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b6b866-2901-4da3-b6ce-3276bf1bcfc6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:20.022074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b6b866-2901-4da3-b6ce-3276bf1bcfc6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:20.024795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b6b866-2901-4da3-b6ce-3276bf1bcfc6 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-e9b6b866-2901-4da3-b6ce-3276bf1bcfc6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:20.039917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:20.041010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:20.042133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:20.042133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:20.042133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:20.042133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:20.042133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 614/1224] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:20.047565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dfd044-c548-4a5d-82d7-0bbfc40fbfc8 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:20.154045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dfd044-c548-4a5d-82d7-0bbfc40fbfc8 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:20.190608 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7f50d0-0575-4343-b25a-7cb8e639feb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:20.191158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7f50d0-0575-4343-b25a-7cb8e639feb4 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:20.191469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7f50d0-0575-4343-b25a-7cb8e639feb4 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:20.191750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7f50d0-0575-4343-b25a-7cb8e639feb4 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:20.192683 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7f50d0-0575-4343-b25a-7cb8e639feb4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:20.192937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7f50d0-0575-4343-b25a-7cb8e639feb4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:20.194479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7f50d0-0575-4343-b25a-7cb8e639feb4 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:20.207888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 611/1225] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:20.215638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b6b866-2901-4da3-b6ce-3276bf1bcfc6 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:20.323081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:20.330766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b6b866-2901-4da3-b6ce-3276bf1bcfc6 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:20.372746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61465be4-8370-4ea1-a31d-0ec3d3a97ab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:20.373356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61465be4-8370-4ea1-a31d-0ec3d3a97ab9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:20.373597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61465be4-8370-4ea1-a31d-0ec3d3a97ab9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:20.373877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61465be4-8370-4ea1-a31d-0ec3d3a97ab9 None admin] PATH_INFO: `/v3/users/placement` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:20.382872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61465be4-8370-4ea1-a31d-0ec3d3a97ab9 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:20.384756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61465be4-8370-4ea1-a31d-0ec3d3a97ab9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-61465be4-8370-4ea1-a31d-0ec3d3a97ab9 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:37:20.396237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:20.397891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:20.399760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 615/1226] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:20 2022] GET /identity/v3/users/placement => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:20.404114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7f50d0-0575-4343-b25a-7cb8e639feb4 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:20.453272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:20.509670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7f50d0-0575-4343-b25a-7cb8e639feb4 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:20.547384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0092c7b-f68b-4053-a9b4-7d5995b5064f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:20.548017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0092c7b-f68b-4053-a9b4-7d5995b5064f None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:20.548304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0092c7b-f68b-4053-a9b4-7d5995b5064f None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:20.548624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0092c7b-f68b-4053-a9b4-7d5995b5064f None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:20.549424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0092c7b-f68b-4053-a9b4-7d5995b5064f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:20.549723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0092c7b-f68b-4053-a9b4-7d5995b5064f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:20.551160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0092c7b-f68b-4053-a9b4-7d5995b5064f None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:20.574551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 612/1227] 10.222.0.40 () {56 vars in 1143 bytes} [Fri Jul 29 04:37:20 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:20.575708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61465be4-8370-4ea1-a31d-0ec3d3a97ab9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:20.684764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61465be4-8370-4ea1-a31d-0ec3d3a97ab9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:20.722424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6b0e4f-1c67-4a2c-ac38-4fea4616d382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:20.722995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6b0e4f-1c67-4a2c-ac38-4fea4616d382 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:20.723281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6b0e4f-1c67-4a2c-ac38-4fea4616d382 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:20.723566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6b0e4f-1c67-4a2c-ac38-4fea4616d382 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:20.734872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6b0e4f-1c67-4a2c-ac38-4fea4616d382 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:20.736341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6b0e4f-1c67-4a2c-ac38-4fea4616d382 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-7e6b0e4f-1c67-4a2c-ac38-4fea4616d382 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:20.751449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:20.752488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:20.753425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:20.753425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:20.753425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:20.753425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:20.753425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 616/1228] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:37:20.759513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0092c7b-f68b-4053-a9b4-7d5995b5064f None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:20.871632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0092c7b-f68b-4053-a9b4-7d5995b5064f None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:20.874201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:20.908791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27487c41-f868-4b9b-8bc3-44c1580c55ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:20.909478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27487c41-f868-4b9b-8bc3-44c1580c55ce None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:20.909733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27487c41-f868-4b9b-8bc3-44c1580c55ce None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:20.909971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27487c41-f868-4b9b-8bc3-44c1580c55ce None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:20.910669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27487c41-f868-4b9b-8bc3-44c1580c55ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:20.910982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27487c41-f868-4b9b-8bc3-44c1580c55ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:20.912363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27487c41-f868-4b9b-8bc3-44c1580c55ce None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:20.925069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 613/1229] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:20.931869 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6b0e4f-1c67-4a2c-ac38-4fea4616d382 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:20.941509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:21.040749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6b0e4f-1c67-4a2c-ac38-4fea4616d382 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:21.076199 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5eee56-bc30-4580-8b56-54620a84aacd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:21.076881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5eee56-bc30-4580-8b56-54620a84aacd None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:21.077170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5eee56-bc30-4580-8b56-54620a84aacd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:21.077472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5eee56-bc30-4580-8b56-54620a84aacd None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:21.084233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5eee56-bc30-4580-8b56-54620a84aacd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:21.085710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5eee56-bc30-4580-8b56-54620a84aacd None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-2a5eee56-bc30-4580-8b56-54620a84aacd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:37:21.097834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:21.098939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:21.100017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:37:21.100017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:37:21.100017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:37:21.100017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:37:21.100017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:37:21.100017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:37:21.100017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:21.100017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:21.100017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 617/1230] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:37:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:21.105358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27487c41-f868-4b9b-8bc3-44c1580c55ce None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:21.215178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27487c41-f868-4b9b-8bc3-44c1580c55ce None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:21.250582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af9a513-1af1-49d2-8c20-f109f12f6e6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:21.251208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9a513-1af1-49d2-8c20-f109f12f6e6d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:21.251627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9a513-1af1-49d2-8c20-f109f12f6e6d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:21.251772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af9a513-1af1-49d2-8c20-f109f12f6e6d None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:21.252530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af9a513-1af1-49d2-8c20-f109f12f6e6d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:21.252879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af9a513-1af1-49d2-8c20-f109f12f6e6d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:21.254308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af9a513-1af1-49d2-8c20-f109f12f6e6d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:21.266927 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 614/1231] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:37:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:37:21.273230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5eee56-bc30-4580-8b56-54620a84aacd None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:21.345132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:21.387280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5eee56-bc30-4580-8b56-54620a84aacd None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:21.425847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201be454-6b92-4d8d-8a1c-67b903e428cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:21.426542 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201be454-6b92-4d8d-8a1c-67b903e428cd None admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:21.426872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201be454-6b92-4d8d-8a1c-67b903e428cd None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:21.427064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201be454-6b92-4d8d-8a1c-67b903e428cd None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/e3da5e428049419a9b35494655f52484/roles/7140c1d0b4f74727a1885faa13c9e6ec` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:21.444015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201be454-6b92-4d8d-8a1c-67b903e428cd None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=e3da5e428049419a9b35494655f52484, role_id=7140c1d0b4f74727a1885faa13c9e6ec)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:21.446576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201be454-6b92-4d8d-8a1c-67b903e428cd None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:21.464217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 618/1232] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:37:21 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/e3da5e428049419a9b35494655f52484/roles/7140c1d0b4f74727a1885faa13c9e6ec => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:37:23.534360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c952faab-310b-4c3d-85ad-9a403f1f2cab None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:23.534894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c952faab-310b-4c3d-85ad-9a403f1f2cab None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:23.535004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c952faab-310b-4c3d-85ad-9a403f1f2cab None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:23.536641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 615/1233] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:37:23.541713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304ab9e-22d8-40cb-8c3c-76f4b7596ee4 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:23.542139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304ab9e-22d8-40cb-8c3c-76f4b7596ee4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:23.542262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f304ab9e-22d8-40cb-8c3c-76f4b7596ee4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:23.561873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f304ab9e-22d8-40cb-8c3c-76f4b7596ee4 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:23.564015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f304ab9e-22d8-40cb-8c3c-76f4b7596ee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:23.637356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 619/1234] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:23 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:37:23.660731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c952faab-310b-4c3d-85ad-9a403f1f2cab None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:23.689956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:23.778755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c952faab-310b-4c3d-85ad-9a403f1f2cab None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:23.815298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a014a7-a71e-4bd6-84ca-ad988448e3fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:23.815958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a014a7-a71e-4bd6-84ca-ad988448e3fb None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:23.816299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a014a7-a71e-4bd6-84ca-ad988448e3fb None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:23.816712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a014a7-a71e-4bd6-84ca-ad988448e3fb None admin] PATH_INFO: `/v3/roles/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:23.828533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a014a7-a71e-4bd6-84ca-ad988448e3fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-71a014a7-a71e-4bd6-84ca-ad988448e3fb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:37:23.834059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:23.835250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:23.837004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 616/1235] 10.222.0.40 () {56 vars in 1102 bytes} [Fri Jul 29 04:37:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:37:23.842501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f304ab9e-22d8-40cb-8c3c-76f4b7596ee4 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:23.892907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:23.951548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f304ab9e-22d8-40cb-8c3c-76f4b7596ee4 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:23.986880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6168aaf9-9c6d-4601-9b52-f73ac3e5c3e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:23.987505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6168aaf9-9c6d-4601-9b52-f73ac3e5c3e1 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:23.987793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6168aaf9-9c6d-4601-9b52-f73ac3e5c3e1 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:23.988074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6168aaf9-9c6d-4601-9b52-f73ac3e5c3e1 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:23.988832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6168aaf9-9c6d-4601-9b52-f73ac3e5c3e1 None admin] RBAC: Adding query filter params (name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:23.989145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6168aaf9-9c6d-4601-9b52-f73ac3e5c3e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:23.990533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6168aaf9-9c6d-4601-9b52-f73ac3e5c3e1 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:24.000956 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 620/1236] 10.222.0.40 () {56 vars in 1103 bytes} [Fri Jul 29 04:37:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:24.006861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a014a7-a71e-4bd6-84ca-ad988448e3fb None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:24.095593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:24.113129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a014a7-a71e-4bd6-84ca-ad988448e3fb None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:24.152793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e69d57-e747-4622-a784-c3528cdb9676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:24.153597 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e69d57-e747-4622-a784-c3528cdb9676 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:24.153937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e69d57-e747-4622-a784-c3528cdb9676 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:24.154279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e69d57-e747-4622-a784-c3528cdb9676 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:24.166622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e69d57-e747-4622-a784-c3528cdb9676 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:24.168238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e69d57-e747-4622-a784-c3528cdb9676 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-77e69d57-e747-4622-a784-c3528cdb9676 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:24.184987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:24.186980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:24.188520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:24.188520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:24.188520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:24.188520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:24.188520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 617/1237] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:24.192625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6168aaf9-9c6d-4601-9b52-f73ac3e5c3e1 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:24.305171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6168aaf9-9c6d-4601-9b52-f73ac3e5c3e1 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:24.317480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:24.347164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18add648-7c90-4786-a438-2646f72171ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:24.347747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18add648-7c90-4786-a438-2646f72171ef None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:24.348064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18add648-7c90-4786-a438-2646f72171ef None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:24.348326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18add648-7c90-4786-a438-2646f72171ef None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:24.349062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18add648-7c90-4786-a438-2646f72171ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:24.349358 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18add648-7c90-4786-a438-2646f72171ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:24.350734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18add648-7c90-4786-a438-2646f72171ef None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:24.364835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 621/1238] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:24.372729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e69d57-e747-4622-a784-c3528cdb9676 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:24.479260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e69d57-e747-4622-a784-c3528cdb9676 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:24.516884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd12fe3-f2df-4be6-81f0-904f0be0a749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:24.517580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd12fe3-f2df-4be6-81f0-904f0be0a749 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:24.517879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd12fe3-f2df-4be6-81f0-904f0be0a749 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:24.518149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd12fe3-f2df-4be6-81f0-904f0be0a749 None admin] PATH_INFO: `/v3/users/placement` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:24.521384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:24.529323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd12fe3-f2df-4be6-81f0-904f0be0a749 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:24.531475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd12fe3-f2df-4be6-81f0-904f0be0a749 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-ecd12fe3-f2df-4be6-81f0-904f0be0a749 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:37:24.545477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:24.547120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:24.548080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 618/1239] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:24 2022] GET /identity/v3/users/placement => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:24.553639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18add648-7c90-4786-a438-2646f72171ef None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:24.668117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18add648-7c90-4786-a438-2646f72171ef None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:24.704376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da403d4f-ec48-4f21-976d-aeadda886512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:24.705213 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da403d4f-ec48-4f21-976d-aeadda886512 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:24.705575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da403d4f-ec48-4f21-976d-aeadda886512 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:24.705856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da403d4f-ec48-4f21-976d-aeadda886512 None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:24.706666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da403d4f-ec48-4f21-976d-aeadda886512 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:24.706995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da403d4f-ec48-4f21-976d-aeadda886512 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:24.708403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da403d4f-ec48-4f21-976d-aeadda886512 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:24.727923 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 622/1240] 10.222.0.40 () {56 vars in 1143 bytes} [Fri Jul 29 04:37:24 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:24.735139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd12fe3-f2df-4be6-81f0-904f0be0a749 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:24.838143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd12fe3-f2df-4be6-81f0-904f0be0a749 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:24.874512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d977d6fd-b2ca-4300-9ab7-d2678aa7862e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:24.875125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977d6fd-b2ca-4300-9ab7-d2678aa7862e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:24.875453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977d6fd-b2ca-4300-9ab7-d2678aa7862e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:24.875692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d977d6fd-b2ca-4300-9ab7-d2678aa7862e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:24.942796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:24.951327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d977d6fd-b2ca-4300-9ab7-d2678aa7862e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:24.953084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d977d6fd-b2ca-4300-9ab7-d2678aa7862e None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:25.061046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 619/1241] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:37:24 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 29 04:37:25.067408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da403d4f-ec48-4f21-976d-aeadda886512 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:25.098307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:25.181166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da403d4f-ec48-4f21-976d-aeadda886512 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:25.217848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361e4c53-46aa-4b37-8c80-f12231523555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:25.218557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361e4c53-46aa-4b37-8c80-f12231523555 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:25.218913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361e4c53-46aa-4b37-8c80-f12231523555 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:25.219246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361e4c53-46aa-4b37-8c80-f12231523555 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:25.230927 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361e4c53-46aa-4b37-8c80-f12231523555 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:25.232826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361e4c53-46aa-4b37-8c80-f12231523555 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-361e4c53-46aa-4b37-8c80-f12231523555 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:25.249543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:25.250701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:25.251782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:25.251782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:25.251782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:25.251782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:25.251782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 623/1242] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:25.258390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d977d6fd-b2ca-4300-9ab7-d2678aa7862e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:25.365016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:25.372052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d977d6fd-b2ca-4300-9ab7-d2678aa7862e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:25.409575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6db830-bb0b-4230-89b6-1671f3b9d415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:25.410317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6db830-bb0b-4230-89b6-1671f3b9d415 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:25.410734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6db830-bb0b-4230-89b6-1671f3b9d415 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:25.411006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6db830-bb0b-4230-89b6-1671f3b9d415 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:25.411821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6db830-bb0b-4230-89b6-1671f3b9d415 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:25.412128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6db830-bb0b-4230-89b6-1671f3b9d415 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:25.413952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6db830-bb0b-4230-89b6-1671f3b9d415 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:25.426987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 620/1243] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:25.433714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361e4c53-46aa-4b37-8c80-f12231523555 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:25.526446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:25.550582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361e4c53-46aa-4b37-8c80-f12231523555 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:25.587990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5830b59a-3f2f-4609-a34f-88515dd146ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:25.588726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5830b59a-3f2f-4609-a34f-88515dd146ed None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:25.588937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5830b59a-3f2f-4609-a34f-88515dd146ed None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:25.589215 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5830b59a-3f2f-4609-a34f-88515dd146ed None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:25.595870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5830b59a-3f2f-4609-a34f-88515dd146ed None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:25.597342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5830b59a-3f2f-4609-a34f-88515dd146ed None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-5830b59a-3f2f-4609-a34f-88515dd146ed None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:37:25.609168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:25.610080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:25.610978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:37:25.610978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:37:25.610978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:37:25.610978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:37:25.610978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:37:25.610978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:37:25.610978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:25.610978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:25.612890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 624/1244] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:37:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:25.616624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6db830-bb0b-4230-89b6-1671f3b9d415 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:25.728548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6db830-bb0b-4230-89b6-1671f3b9d415 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:25.767302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede22b03-95df-4e3a-a63d-b42b32b7df26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:25.768029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede22b03-95df-4e3a-a63d-b42b32b7df26 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:25.768326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede22b03-95df-4e3a-a63d-b42b32b7df26 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:25.768620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede22b03-95df-4e3a-a63d-b42b32b7df26 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:25.769684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede22b03-95df-4e3a-a63d-b42b32b7df26 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:25.770275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede22b03-95df-4e3a-a63d-b42b32b7df26 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:25.772399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede22b03-95df-4e3a-a63d-b42b32b7df26 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:25.786576 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 621/1245] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:37:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:37:25.793237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5830b59a-3f2f-4609-a34f-88515dd146ed None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:25.902563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5830b59a-3f2f-4609-a34f-88515dd146ed None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:25.940396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52871a6b-0f67-4d86-b77a-a9220a8c0820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:25.941127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52871a6b-0f67-4d86-b77a-a9220a8c0820 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:25.941406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52871a6b-0f67-4d86-b77a-a9220a8c0820 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:25.941681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52871a6b-0f67-4d86-b77a-a9220a8c0820 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:25.942465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52871a6b-0f67-4d86-b77a-a9220a8c0820 None admin] RBAC: Adding query filter params (role.id=7140c1d0b4f74727a1885faa13c9e6ec, scope.project.id=441bcfe715d44efea69d556677094857, user.id=e3da5e428049419a9b35494655f52484) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:25.942772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52871a6b-0f67-4d86-b77a-a9220a8c0820 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:25.944208 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52871a6b-0f67-4d86-b77a-a9220a8c0820 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:25.951086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 625/1246] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:37:25 2022] GET /identity/v3/role_assignments?user.id=e3da5e428049419a9b35494655f52484&scope.project.id=441bcfe715d44efea69d556677094857&role.id=7140c1d0b4f74727a1885faa13c9e6ec => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:37:28.088710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cbc7f4-2ca8-43a5-b7a6-e7b0af09d9a5 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:28.089290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cbc7f4-2ca8-43a5-b7a6-e7b0af09d9a5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:28.089466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10cbc7f4-2ca8-43a5-b7a6-e7b0af09d9a5 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:28.091257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 622/1247] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:37:28.096617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42242853-2fcc-40cd-ade2-841fb1ace52a None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:28.097032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42242853-2fcc-40cd-ade2-841fb1ace52a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:28.097217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42242853-2fcc-40cd-ade2-841fb1ace52a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:28.104732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:28.119638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-42242853-2fcc-40cd-ade2-841fb1ace52a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:28.121045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-42242853-2fcc-40cd-ade2-841fb1ace52a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:28.200931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 626/1248] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:28 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:37:28.225841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cbc7f4-2ca8-43a5-b7a6-e7b0af09d9a5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:28.246849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:28.346148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10cbc7f4-2ca8-43a5-b7a6-e7b0af09d9a5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:28.384278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721af983-3088-4fe5-b58b-2d78e01a6ba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:28.384952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721af983-3088-4fe5-b58b-2d78e01a6ba6 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:28.385259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721af983-3088-4fe5-b58b-2d78e01a6ba6 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:28.385669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721af983-3088-4fe5-b58b-2d78e01a6ba6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:28.397311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721af983-3088-4fe5-b58b-2d78e01a6ba6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-721af983-3088-4fe5-b58b-2d78e01a6ba6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:37:28.403091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:28.404108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:28.404997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 623/1249] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:37:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:28.410781 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42242853-2fcc-40cd-ade2-841fb1ace52a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:28.522214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42242853-2fcc-40cd-ade2-841fb1ace52a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:28.561501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:28.567927 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e651974a-09a6-46ae-88cc-4d39d26b9491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:28.568621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e651974a-09a6-46ae-88cc-4d39d26b9491 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:28.569038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e651974a-09a6-46ae-88cc-4d39d26b9491 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:28.569448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e651974a-09a6-46ae-88cc-4d39d26b9491 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:28.570441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e651974a-09a6-46ae-88cc-4d39d26b9491 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:28.571057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e651974a-09a6-46ae-88cc-4d39d26b9491 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:28.572901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e651974a-09a6-46ae-88cc-4d39d26b9491 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:28.583037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 627/1250] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:37:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:37:28.589852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721af983-3088-4fe5-b58b-2d78e01a6ba6 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:28.685522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:28.715900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721af983-3088-4fe5-b58b-2d78e01a6ba6 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:28.757565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e92d600-5d52-4e74-af4f-426e007fea0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:28.759317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92d600-5d52-4e74-af4f-426e007fea0d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:28.759690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92d600-5d52-4e74-af4f-426e007fea0d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:28.759982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92d600-5d52-4e74-af4f-426e007fea0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:28.771582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e92d600-5d52-4e74-af4f-426e007fea0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:28.773100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e92d600-5d52-4e74-af4f-426e007fea0d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-2e92d600-5d52-4e74-af4f-426e007fea0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:28.788154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:28.789337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:28.790407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:28.790407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:28.790407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:28.790407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:28.790407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 624/1251] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:28.796005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e651974a-09a6-46ae-88cc-4d39d26b9491 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:28.914461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e651974a-09a6-46ae-88cc-4d39d26b9491 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:28.962779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff229462-2cca-4a69-87c4-fab7678fe66e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:28.963517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff229462-2cca-4a69-87c4-fab7678fe66e None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:28.963809 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff229462-2cca-4a69-87c4-fab7678fe66e None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:28.964086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff229462-2cca-4a69-87c4-fab7678fe66e None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:28.964809 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff229462-2cca-4a69-87c4-fab7678fe66e None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:28.965115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff229462-2cca-4a69-87c4-fab7678fe66e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:28.966651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff229462-2cca-4a69-87c4-fab7678fe66e None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:28.980481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 628/1252] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:28.986440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e92d600-5d52-4e74-af4f-426e007fea0d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:29.100434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e92d600-5d52-4e74-af4f-426e007fea0d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:29.127805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:29.136361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebe9c59-9f38-4342-af2e-7e81942748b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:29.137024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebe9c59-9f38-4342-af2e-7e81942748b0 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:29.137352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebe9c59-9f38-4342-af2e-7e81942748b0 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:29.137661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebe9c59-9f38-4342-af2e-7e81942748b0 None admin] PATH_INFO: `/v3/users/placement` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:29.145875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebe9c59-9f38-4342-af2e-7e81942748b0 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:29.147344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebe9c59-9f38-4342-af2e-7e81942748b0 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-8ebe9c59-9f38-4342-af2e-7e81942748b0 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:37:29.158791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:29.160149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:29.161191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 625/1253] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:28 2022] GET /identity/v3/users/placement => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 29 04:37:29.166789 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff229462-2cca-4a69-87c4-fab7678fe66e None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:29.229369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:29.280394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff229462-2cca-4a69-87c4-fab7678fe66e None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:29.320964 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4965899d-c2d4-425d-aa8d-347ea604f7ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:29.321690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4965899d-c2d4-425d-aa8d-347ea604f7ab None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:29.322049 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4965899d-c2d4-425d-aa8d-347ea604f7ab None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:29.322302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4965899d-c2d4-425d-aa8d-347ea604f7ab None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:29.323237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4965899d-c2d4-425d-aa8d-347ea604f7ab None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:29.323589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4965899d-c2d4-425d-aa8d-347ea604f7ab None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:29.325186 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4965899d-c2d4-425d-aa8d-347ea604f7ab None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:29.348453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 629/1254] 10.222.0.40 () {56 vars in 1143 bytes} [Fri Jul 29 04:37:29 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:29.349699 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebe9c59-9f38-4342-af2e-7e81942748b0 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:29.463951 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebe9c59-9f38-4342-af2e-7e81942748b0 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:29.502628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea312f1-9af0-4ef6-8d54-d7aef99af55a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:29.503328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea312f1-9af0-4ef6-8d54-d7aef99af55a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:29.503624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea312f1-9af0-4ef6-8d54-d7aef99af55a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:29.503904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea312f1-9af0-4ef6-8d54-d7aef99af55a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:29.563704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:29.578669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea312f1-9af0-4ef6-8d54-d7aef99af55a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:29.580171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea312f1-9af0-4ef6-8d54-d7aef99af55a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:29.690225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 626/1255] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:37:29 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:37:29.696337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4965899d-c2d4-425d-aa8d-347ea604f7ab None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:29.798022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4965899d-c2d4-425d-aa8d-347ea604f7ab None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:29.813438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:29.835664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedf417a-4098-49ae-b357-ba402e70c458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:29.836456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedf417a-4098-49ae-b357-ba402e70c458 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:29.836796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedf417a-4098-49ae-b357-ba402e70c458 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:29.837079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedf417a-4098-49ae-b357-ba402e70c458 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:29.847249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedf417a-4098-49ae-b357-ba402e70c458 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:29.848749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedf417a-4098-49ae-b357-ba402e70c458 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-aedf417a-4098-49ae-b357-ba402e70c458 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:29.864851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:29.866161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:29.867837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:29.867837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:29.867837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:29.867837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:29.871495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 630/1256] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:29.872637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea312f1-9af0-4ef6-8d54-d7aef99af55a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:29.987132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:30.002222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea312f1-9af0-4ef6-8d54-d7aef99af55a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:30.041014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91feac24-7b32-44d5-8d8d-56688aed335f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:30.041631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91feac24-7b32-44d5-8d8d-56688aed335f None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:30.041912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91feac24-7b32-44d5-8d8d-56688aed335f None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:30.042185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91feac24-7b32-44d5-8d8d-56688aed335f None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:30.042876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91feac24-7b32-44d5-8d8d-56688aed335f None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:30.043187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91feac24-7b32-44d5-8d8d-56688aed335f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:30.044547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91feac24-7b32-44d5-8d8d-56688aed335f None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:30.056301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 627/1257] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:37:30.061851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedf417a-4098-49ae-b357-ba402e70c458 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:30.178091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedf417a-4098-49ae-b357-ba402e70c458 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:30.217880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadc6c27-f690-4088-a7ca-7c0fdbe0c639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:30.218510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadc6c27-f690-4088-a7ca-7c0fdbe0c639 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:30.218847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadc6c27-f690-4088-a7ca-7c0fdbe0c639 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:30.219071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadc6c27-f690-4088-a7ca-7c0fdbe0c639 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:30.227110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadc6c27-f690-4088-a7ca-7c0fdbe0c639 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:30.228746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadc6c27-f690-4088-a7ca-7c0fdbe0c639 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-fadc6c27-f690-4088-a7ca-7c0fdbe0c639 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:37:30.243506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:30.244663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:30.245703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:37:30.245703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:37:30.245703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:37:30.245703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:37:30.245703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:37:30.245703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:37:30.245703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:30.245703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:30.245703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 631/1258] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:37:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:30.251302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91feac24-7b32-44d5-8d8d-56688aed335f None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:30.361111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91feac24-7b32-44d5-8d8d-56688aed335f None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:30.405091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae6f632-de1b-4bff-b44c-c97fc438fba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:30.406151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae6f632-de1b-4bff-b44c-c97fc438fba3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:30.406633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae6f632-de1b-4bff-b44c-c97fc438fba3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:30.407164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae6f632-de1b-4bff-b44c-c97fc438fba3 None admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:30.408581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae6f632-de1b-4bff-b44c-c97fc438fba3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:30.409104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae6f632-de1b-4bff-b44c-c97fc438fba3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:30.411158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae6f632-de1b-4bff-b44c-c97fc438fba3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:30.431837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 628/1259] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:37:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:30.439112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadc6c27-f690-4088-a7ca-7c0fdbe0c639 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:30.469028 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:30.554028 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadc6c27-f690-4088-a7ca-7c0fdbe0c639 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:30.595779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476bcc7d-2ab3-4fbc-abb4-9fc21c465634 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:30.596629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476bcc7d-2ab3-4fbc-abb4-9fc21c465634 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:30.596877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476bcc7d-2ab3-4fbc-abb4-9fc21c465634 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:30.597269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476bcc7d-2ab3-4fbc-abb4-9fc21c465634 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:30.598145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476bcc7d-2ab3-4fbc-abb4-9fc21c465634 None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=441bcfe715d44efea69d556677094857, user.id=e3da5e428049419a9b35494655f52484) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:30.598520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476bcc7d-2ab3-4fbc-abb4-9fc21c465634 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:30.600331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476bcc7d-2ab3-4fbc-abb4-9fc21c465634 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:30.607331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 632/1260] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:37:30 2022] GET /identity/v3/role_assignments?user.id=e3da5e428049419a9b35494655f52484&scope.project.id=441bcfe715d44efea69d556677094857&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:37:32.610828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba25475-be1e-4df9-b486-3d868574f4f1 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:32.611363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba25475-be1e-4df9-b486-3d868574f4f1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:32.611363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba25475-be1e-4df9-b486-3d868574f4f1 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:32.612749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 629/1261] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:37:32.617793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c85b93-00fa-47a6-bb9e-397581356a84 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:32.618211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c85b93-00fa-47a6-bb9e-397581356a84 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:32.618361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7c85b93-00fa-47a6-bb9e-397581356a84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:32.636639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c7c85b93-00fa-47a6-bb9e-397581356a84 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:32.637825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c7c85b93-00fa-47a6-bb9e-397581356a84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:32.703341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 633/1262] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:32 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:37:32.725631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba25475-be1e-4df9-b486-3d868574f4f1 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:32.736425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:32.825444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba25475-be1e-4df9-b486-3d868574f4f1 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:32.858219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07273a62-3722-4841-92b7-e55d6693cd5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:32.858862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07273a62-3722-4841-92b7-e55d6693cd5e None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:32.859143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07273a62-3722-4841-92b7-e55d6693cd5e None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:32.859419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07273a62-3722-4841-92b7-e55d6693cd5e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:32.869367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07273a62-3722-4841-92b7-e55d6693cd5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-07273a62-3722-4841-92b7-e55d6693cd5e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:37:32.874194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:32.875188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:32.876015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 630/1263] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:37:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:32.881593 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c85b93-00fa-47a6-bb9e-397581356a84 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:32.901043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:32.979579 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7c85b93-00fa-47a6-bb9e-397581356a84 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:33.012930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9872c373-8311-4af2-bb91-e48a1283edf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:33.013537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9872c373-8311-4af2-bb91-e48a1283edf2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:33.013820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9872c373-8311-4af2-bb91-e48a1283edf2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:33.014097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9872c373-8311-4af2-bb91-e48a1283edf2 None admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:33.014807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9872c373-8311-4af2-bb91-e48a1283edf2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:33.015113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9872c373-8311-4af2-bb91-e48a1283edf2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:33.016470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9872c373-8311-4af2-bb91-e48a1283edf2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:33.024223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 634/1264] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:37:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:33.029891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07273a62-3722-4841-92b7-e55d6693cd5e None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:33.096804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:33.129196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07273a62-3722-4841-92b7-e55d6693cd5e None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:33.161889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb44a19e-a7d5-4360-8cde-0acb47ea5721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:33.162513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb44a19e-a7d5-4360-8cde-0acb47ea5721 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:33.162794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb44a19e-a7d5-4360-8cde-0acb47ea5721 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:33.163069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb44a19e-a7d5-4360-8cde-0acb47ea5721 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:33.173084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb44a19e-a7d5-4360-8cde-0acb47ea5721 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:33.174517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb44a19e-a7d5-4360-8cde-0acb47ea5721 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-fb44a19e-a7d5-4360-8cde-0acb47ea5721 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:33.188985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:33.190020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:33.191131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:33.191131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:33.191131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:33.191131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:33.191131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 631/1265] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:33.196260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9872c373-8311-4af2-bb91-e48a1283edf2 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:33.286633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:33.314701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9872c373-8311-4af2-bb91-e48a1283edf2 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:33.355222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c438a99d-5f96-477d-8e63-b673086399d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:33.355840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438a99d-5f96-477d-8e63-b673086399d4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:33.356127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438a99d-5f96-477d-8e63-b673086399d4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:33.356408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438a99d-5f96-477d-8e63-b673086399d4 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:33.357133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c438a99d-5f96-477d-8e63-b673086399d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:33.357445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c438a99d-5f96-477d-8e63-b673086399d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:33.358819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c438a99d-5f96-477d-8e63-b673086399d4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:33.371018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 635/1266] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:33.376736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb44a19e-a7d5-4360-8cde-0acb47ea5721 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:33.472651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb44a19e-a7d5-4360-8cde-0acb47ea5721 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:33.492425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:33.505999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acac38f-8d17-46ca-856d-b42b81d0b92a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:33.506625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acac38f-8d17-46ca-856d-b42b81d0b92a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:33.506906 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acac38f-8d17-46ca-856d-b42b81d0b92a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:33.507238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acac38f-8d17-46ca-856d-b42b81d0b92a None admin] PATH_INFO: `/v3/users/placement` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:33.515907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acac38f-8d17-46ca-856d-b42b81d0b92a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:33.517374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acac38f-8d17-46ca-856d-b42b81d0b92a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-0acac38f-8d17-46ca-856d-b42b81d0b92a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:37:33.528423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:33.529536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:33.530660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 632/1267] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:33 2022] GET /identity/v3/users/placement => generated 87 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:33.535414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c438a99d-5f96-477d-8e63-b673086399d4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:33.633088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c438a99d-5f96-477d-8e63-b673086399d4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:33.666883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6e7dac-0947-4db9-bc6a-fff6ad97a2af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:33.667501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6e7dac-0947-4db9-bc6a-fff6ad97a2af None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:33.667788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6e7dac-0947-4db9-bc6a-fff6ad97a2af None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:33.668070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6e7dac-0947-4db9-bc6a-fff6ad97a2af None admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:33.668901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6e7dac-0947-4db9-bc6a-fff6ad97a2af None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:33.669278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6e7dac-0947-4db9-bc6a-fff6ad97a2af None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:33.670977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6e7dac-0947-4db9-bc6a-fff6ad97a2af None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:33.680280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:33.688976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 636/1268] 10.222.0.40 () {56 vars in 1143 bytes} [Fri Jul 29 04:37:33 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:33.694582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acac38f-8d17-46ca-856d-b42b81d0b92a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:33.791150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acac38f-8d17-46ca-856d-b42b81d0b92a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:33.823585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e6fe08-0d5d-4a49-bd6b-f14fe46b9e6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:33.824210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e6fe08-0d5d-4a49-bd6b-f14fe46b9e6a None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:33.824493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e6fe08-0d5d-4a49-bd6b-f14fe46b9e6a None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:33.824794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e6fe08-0d5d-4a49-bd6b-f14fe46b9e6a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:33.835242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e6fe08-0d5d-4a49-bd6b-f14fe46b9e6a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:33.837311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e6fe08-0d5d-4a49-bd6b-f14fe46b9e6a None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-e8e6fe08-0d5d-4a49-bd6b-f14fe46b9e6a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:33.850898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:33.853458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:33.854569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:33.854569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:33.854569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:33.854569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:33.854569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 633/1269] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:33.858240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6e7dac-0947-4db9-bc6a-fff6ad97a2af None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:33.955553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6e7dac-0947-4db9-bc6a-fff6ad97a2af None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:33.989769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cd8d23-dcb0-4d62-bb34-f7009dc314dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:33.990390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cd8d23-dcb0-4d62-bb34-f7009dc314dc None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:33.990678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cd8d23-dcb0-4d62-bb34-f7009dc314dc None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:33.990961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1cd8d23-dcb0-4d62-bb34-f7009dc314dc None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:33.991674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cd8d23-dcb0-4d62-bb34-f7009dc314dc None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:33.992130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cd8d23-dcb0-4d62-bb34-f7009dc314dc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:33.993537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1cd8d23-dcb0-4d62-bb34-f7009dc314dc None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:34.004656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 637/1270] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:34.010446 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e6fe08-0d5d-4a49-bd6b-f14fe46b9e6a None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:34.042137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:34.109794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e6fe08-0d5d-4a49-bd6b-f14fe46b9e6a None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:34.142217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13858f03-915d-4c7d-a278-8e23acf426ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:34.142845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13858f03-915d-4c7d-a278-8e23acf426ef None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:34.143129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13858f03-915d-4c7d-a278-8e23acf426ef None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:34.143407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13858f03-915d-4c7d-a278-8e23acf426ef None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:34.150490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13858f03-915d-4c7d-a278-8e23acf426ef None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:34.151928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13858f03-915d-4c7d-a278-8e23acf426ef None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-13858f03-915d-4c7d-a278-8e23acf426ef None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:37:34.162972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:34.164155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:34.165511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:37:34.165511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:37:34.165511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:37:34.165511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:37:34.165511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:37:34.165511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:37:34.165511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:34.165511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:34.165511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 634/1271] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:37:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:34.169842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cd8d23-dcb0-4d62-bb34-f7009dc314dc None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:34.227439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:34.269699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1cd8d23-dcb0-4d62-bb34-f7009dc314dc None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:34.304487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad96384c-18c9-42c1-acf0-44dff20bb1b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:34.305063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad96384c-18c9-42c1-acf0-44dff20bb1b0 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:34.305350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad96384c-18c9-42c1-acf0-44dff20bb1b0 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:34.305625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad96384c-18c9-42c1-acf0-44dff20bb1b0 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:34.306371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad96384c-18c9-42c1-acf0-44dff20bb1b0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:34.306711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad96384c-18c9-42c1-acf0-44dff20bb1b0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:34.308148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad96384c-18c9-42c1-acf0-44dff20bb1b0 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:34.321175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 638/1272] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:37:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:34.327183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13858f03-915d-4c7d-a278-8e23acf426ef None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:34.416492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:34.425451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13858f03-915d-4c7d-a278-8e23acf426ef None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:34.458976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcc7c4a-cb52-4e7c-980b-a77fdd5f6cc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:34.459631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc7c4a-cb52-4e7c-980b-a77fdd5f6cc2 None admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:34.459920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc7c4a-cb52-4e7c-980b-a77fdd5f6cc2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:34.460241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc7c4a-cb52-4e7c-980b-a77fdd5f6cc2 None admin] PATH_INFO: `/v3/projects/441bcfe715d44efea69d556677094857/users/e3da5e428049419a9b35494655f52484/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:34.462205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcc7c4a-cb52-4e7c-980b-a77fdd5f6cc2 None admin] RBAC: Authorizing `identity:create_grant(project_id=441bcfe715d44efea69d556677094857, user_id=e3da5e428049419a9b35494655f52484, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:34.463620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcc7c4a-cb52-4e7c-980b-a77fdd5f6cc2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:34.475712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 635/1273] 10.222.0.40 () {58 vars in 1439 bytes} [Fri Jul 29 04:37:34 2022] PUT /identity/v3/projects/441bcfe715d44efea69d556677094857/users/e3da5e428049419a9b35494655f52484/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:37:36.437205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dad3ed-cd1b-47ec-917c-9cdab75e0098 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:36.437758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dad3ed-cd1b-47ec-917c-9cdab75e0098 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:36.437758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dad3ed-cd1b-47ec-917c-9cdab75e0098 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:36.439117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 639/1274] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:37:36.444076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4295f5-e0d3-4569-b8d5-ceada2b8cd62 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:36.444533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4295f5-e0d3-4569-b8d5-ceada2b8cd62 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:36.444692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4295f5-e0d3-4569-b8d5-ceada2b8cd62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:36.462533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-1a4295f5-e0d3-4569-b8d5-ceada2b8cd62 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:36.463741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-1a4295f5-e0d3-4569-b8d5-ceada2b8cd62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:36.529292 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 636/1275] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:36 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:37:36.552671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dad3ed-cd1b-47ec-917c-9cdab75e0098 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:36.650997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dad3ed-cd1b-47ec-917c-9cdab75e0098 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:36.668830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:36.687197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d921110-5a67-47df-b65d-b9798006c0ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:36.687891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d921110-5a67-47df-b65d-b9798006c0ae None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:36.688126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d921110-5a67-47df-b65d-b9798006c0ae None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:36.688433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d921110-5a67-47df-b65d-b9798006c0ae None admin] PATH_INFO: `/v3/roles/admin` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:36.698459 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d921110-5a67-47df-b65d-b9798006c0ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-0d921110-5a67-47df-b65d-b9798006c0ae None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 29 04:37:36.703367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:36.704465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:36.705485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 640/1276] 10.222.0.40 () {56 vars in 1096 bytes} [Fri Jul 29 04:37:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:36.711127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4295f5-e0d3-4569-b8d5-ceada2b8cd62 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:36.787946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:36.807255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4295f5-e0d3-4569-b8d5-ceada2b8cd62 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:36.840748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b35e03e-cf17-4574-b3d3-e24ac8741740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:36.841364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b35e03e-cf17-4574-b3d3-e24ac8741740 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:36.841652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b35e03e-cf17-4574-b3d3-e24ac8741740 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:36.841933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b35e03e-cf17-4574-b3d3-e24ac8741740 None admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:36.842652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b35e03e-cf17-4574-b3d3-e24ac8741740 None admin] RBAC: Adding query filter params (name=admin) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:36.842963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b35e03e-cf17-4574-b3d3-e24ac8741740 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:36.844333 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b35e03e-cf17-4574-b3d3-e24ac8741740 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:36.852852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 637/1277] 10.222.0.40 () {56 vars in 1099 bytes} [Fri Jul 29 04:37:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:36.858528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d921110-5a67-47df-b65d-b9798006c0ae None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:36.962261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d921110-5a67-47df-b65d-b9798006c0ae None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:36.995333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d781e5c9-8f73-4be5-8590-ad070b4b2cb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:36.995959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d781e5c9-8f73-4be5-8590-ad070b4b2cb2 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:36.996241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d781e5c9-8f73-4be5-8590-ad070b4b2cb2 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:36.996515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d781e5c9-8f73-4be5-8590-ad070b4b2cb2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:37.006235 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d781e5c9-8f73-4be5-8590-ad070b4b2cb2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:37.007933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d781e5c9-8f73-4be5-8590-ad070b4b2cb2 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-d781e5c9-8f73-4be5-8590-ad070b4b2cb2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:37.022056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:37.023612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:37.024693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:37.024693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:37.024693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:37.024693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:37.024693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 641/1278] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:37.029913 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b35e03e-cf17-4574-b3d3-e24ac8741740 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:37.127489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b35e03e-cf17-4574-b3d3-e24ac8741740 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:37.160360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947efe74-c751-42f8-b10c-69180d508fc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:37.161003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947efe74-c751-42f8-b10c-69180d508fc2 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:37.161291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947efe74-c751-42f8-b10c-69180d508fc2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:37.161573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947efe74-c751-42f8-b10c-69180d508fc2 None admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:37.162267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947efe74-c751-42f8-b10c-69180d508fc2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:37.162578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947efe74-c751-42f8-b10c-69180d508fc2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:37.164555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947efe74-c751-42f8-b10c-69180d508fc2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:37.176190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 638/1279] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:37.182415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d781e5c9-8f73-4be5-8590-ad070b4b2cb2 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:37.212172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:37.284094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d781e5c9-8f73-4be5-8590-ad070b4b2cb2 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:37.317201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b4b330-dee8-40d1-b9ac-ba397f14a030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:37.317823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b4b330-dee8-40d1-b9ac-ba397f14a030 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:37.318103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b4b330-dee8-40d1-b9ac-ba397f14a030 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:37.318379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b4b330-dee8-40d1-b9ac-ba397f14a030 None admin] PATH_INFO: `/v3/users/placement` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:37.325777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b4b330-dee8-40d1-b9ac-ba397f14a030 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:37.327202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b4b330-dee8-40d1-b9ac-ba397f14a030 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-69b4b330-dee8-40d1-b9ac-ba397f14a030 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 29 04:37:37.338296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:37.339395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:37.340351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 642/1280] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:37 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:37.345185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947efe74-c751-42f8-b10c-69180d508fc2 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:37.348888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:37.442206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947efe74-c751-42f8-b10c-69180d508fc2 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:37.475441 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a8eb9a-992d-4181-a191-6e40f58d8786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:37.477056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a8eb9a-992d-4181-a191-6e40f58d8786 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:37.477353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a8eb9a-992d-4181-a191-6e40f58d8786 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:37.477639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a8eb9a-992d-4181-a191-6e40f58d8786 None admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:37.478434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a8eb9a-992d-4181-a191-6e40f58d8786 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:37.478777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a8eb9a-992d-4181-a191-6e40f58d8786 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:37.480189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29a8eb9a-992d-4181-a191-6e40f58d8786 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:37.496866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 639/1281] 10.222.0.40 () {56 vars in 1143 bytes} [Fri Jul 29 04:37:37 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:37.502522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b4b330-dee8-40d1-b9ac-ba397f14a030 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:37.592357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:37.602997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b4b330-dee8-40d1-b9ac-ba397f14a030 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:37.635300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bceb5ecd-d41b-4011-b425-100b382a23e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:37.635911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bceb5ecd-d41b-4011-b425-100b382a23e9 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:37.636200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bceb5ecd-d41b-4011-b425-100b382a23e9 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:37.636483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bceb5ecd-d41b-4011-b425-100b382a23e9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:37.703648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bceb5ecd-d41b-4011-b425-100b382a23e9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:37.705183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bceb5ecd-d41b-4011-b425-100b382a23e9 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:37.790058 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:37.800711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 643/1282] 10.222.0.40 () {58 vars in 1282 bytes} [Fri Jul 29 04:37:37 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 29 04:37:37.806407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a8eb9a-992d-4181-a191-6e40f58d8786 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:37.865901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:37.903778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29a8eb9a-992d-4181-a191-6e40f58d8786 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:37.936372 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19ce0a-45bf-46b0-a34c-7dd73699852d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:37.937028 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19ce0a-45bf-46b0-a34c-7dd73699852d None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:37.937314 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19ce0a-45bf-46b0-a34c-7dd73699852d None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:37.937591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19ce0a-45bf-46b0-a34c-7dd73699852d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:37.947745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c19ce0a-45bf-46b0-a34c-7dd73699852d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:37.949216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c19ce0a-45bf-46b0-a34c-7dd73699852d None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-8c19ce0a-45bf-46b0-a34c-7dd73699852d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 29 04:37:37.963980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:37.965042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:37.966162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 29 04:37:37.966162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 29 04:37:37.966162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 29 04:37:37.966162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:37.966162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 640/1283] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:37:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:37.970949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bceb5ecd-d41b-4011-b425-100b382a23e9 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:38.066297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bceb5ecd-d41b-4011-b425-100b382a23e9 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:38.099913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dd55aa-1550-4af7-8db0-b580527f2b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:38.100523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dd55aa-1550-4af7-8db0-b580527f2b94 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:38.100902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dd55aa-1550-4af7-8db0-b580527f2b94 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:38.101180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dd55aa-1550-4af7-8db0-b580527f2b94 None admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:38.101868 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dd55aa-1550-4af7-8db0-b580527f2b94 None admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:38.102174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dd55aa-1550-4af7-8db0-b580527f2b94 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:38.103534 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36dd55aa-1550-4af7-8db0-b580527f2b94 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:38.114380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 644/1284] 10.222.0.40 () {56 vars in 1109 bytes} [Fri Jul 29 04:37:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:38.121010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19ce0a-45bf-46b0-a34c-7dd73699852d None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:38.219263 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19ce0a-45bf-46b0-a34c-7dd73699852d None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:38.232468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:38.254717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58547af-f3fa-4b9d-bdce-8c786d417f54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:38.255518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58547af-f3fa-4b9d-bdce-8c786d417f54 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:38.255825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58547af-f3fa-4b9d-bdce-8c786d417f54 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:38.256112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58547af-f3fa-4b9d-bdce-8c786d417f54 None admin] PATH_INFO: `/v3/projects/service` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:38.265235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58547af-f3fa-4b9d-bdce-8c786d417f54 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:38.267113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58547af-f3fa-4b9d-bdce-8c786d417f54 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application [None req-e58547af-f3fa-4b9d-bdce-8c786d417f54 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 29 04:37:38.280686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:38.281750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:38.282850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 29 04:37:38.282850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 29 04:37:38.282850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 29 04:37:38.282850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 29 04:37:38.282850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 29 04:37:38.282850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 29 04:37:38.282850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 29 04:37:38.282850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: ERROR keystone.server.flask.application Jul 29 04:37:38.282850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 641/1285] 10.222.0.40 () {56 vars in 1111 bytes} [Fri Jul 29 04:37:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:38.287612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dd55aa-1550-4af7-8db0-b580527f2b94 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:38.337543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:38.386453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36dd55aa-1550-4af7-8db0-b580527f2b94 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:38.418385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234cc9f6-85c6-457a-b9b7-1ffd21be9b73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:38.418997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234cc9f6-85c6-457a-b9b7-1ffd21be9b73 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:38.419286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234cc9f6-85c6-457a-b9b7-1ffd21be9b73 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:38.419570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234cc9f6-85c6-457a-b9b7-1ffd21be9b73 None admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:38.420578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234cc9f6-85c6-457a-b9b7-1ffd21be9b73 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:38.420925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234cc9f6-85c6-457a-b9b7-1ffd21be9b73 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:38.422353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234cc9f6-85c6-457a-b9b7-1ffd21be9b73 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:38.434421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 645/1286] 10.222.0.40 () {56 vars in 1148 bytes} [Fri Jul 29 04:37:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:38.440364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58547af-f3fa-4b9d-bdce-8c786d417f54 None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:38.537553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58547af-f3fa-4b9d-bdce-8c786d417f54 None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:38.571172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e5c795-fd45-42a8-a059-afa77023a0fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:38.571787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e5c795-fd45-42a8-a059-afa77023a0fe None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:38.572077 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e5c795-fd45-42a8-a059-afa77023a0fe None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:38.572361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e5c795-fd45-42a8-a059-afa77023a0fe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:38.573178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e5c795-fd45-42a8-a059-afa77023a0fe None admin] RBAC: Adding query filter params (role.id=ca6111d7d86f4ac29c63630ce5250fd1, scope.project.id=441bcfe715d44efea69d556677094857, user.id=e3da5e428049419a9b35494655f52484) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:38.573497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e5c795-fd45-42a8-a059-afa77023a0fe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:38.574929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e5c795-fd45-42a8-a059-afa77023a0fe None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:38.580640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 642/1287] 10.222.0.40 () {56 vars in 1374 bytes} [Fri Jul 29 04:37:38 2022] GET /identity/v3/role_assignments?user.id=e3da5e428049419a9b35494655f52484&scope.project.id=441bcfe715d44efea69d556677094857&role.id=ca6111d7d86f4ac29c63630ce5250fd1 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:40.611999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fa76b-c4fc-458a-b4a3-3eb5553d18d3 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:40.612531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fa76b-c4fc-458a-b4a3-3eb5553d18d3 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:40.612531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6fa76b-c4fc-458a-b4a3-3eb5553d18d3 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:40.613928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 646/1288] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:37:40.619106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadfe8dc-bfdf-43ad-8aea-fda0e080f81e None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:40.619530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadfe8dc-bfdf-43ad-8aea-fda0e080f81e None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:40.619648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aadfe8dc-bfdf-43ad-8aea-fda0e080f81e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:40.637499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-aadfe8dc-bfdf-43ad-8aea-fda0e080f81e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:40.638684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-aadfe8dc-bfdf-43ad-8aea-fda0e080f81e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:40.668497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:40.705231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 643/1289] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:40 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:37:40.728359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6fa76b-c4fc-458a-b4a3-3eb5553d18d3 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:40.826450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6fa76b-c4fc-458a-b4a3-3eb5553d18d3 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:40.837246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:40.860067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ec4b22-21ed-4c56-8c8f-4c82f30841a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:40.860729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec4b22-21ed-4c56-8c8f-4c82f30841a4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:40.861017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec4b22-21ed-4c56-8c8f-4c82f30841a4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:40.861297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ec4b22-21ed-4c56-8c8f-4c82f30841a4 None admin] PATH_INFO: `/v3/services/placement` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:40.861918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ec4b22-21ed-4c56-8c8f-4c82f30841a4 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:40.863332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ec4b22-21ed-4c56-8c8f-4c82f30841a4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-c9ec4b22-21ed-4c56-8c8f-4c82f30841a4 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:40.871951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:40.873350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 29 04:37:40.874305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:40.876795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 647/1290] 10.222.0.40 () {56 vars in 1117 bytes} [Fri Jul 29 04:37:40 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:40.879085 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadfe8dc-bfdf-43ad-8aea-fda0e080f81e None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:40.977487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aadfe8dc-bfdf-43ad-8aea-fda0e080f81e None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:41.010507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4148c7-8234-408f-a43a-b885759ac7f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:41.011119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4148c7-8234-408f-a43a-b885759ac7f1 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:41.011408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4148c7-8234-408f-a43a-b885759ac7f1 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:41.011688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4148c7-8234-408f-a43a-b885759ac7f1 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:41.012364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4148c7-8234-408f-a43a-b885759ac7f1 None admin] RBAC: Adding query filter params (name=placement) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:41.012699 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4148c7-8234-408f-a43a-b885759ac7f1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:41.014060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4148c7-8234-408f-a43a-b885759ac7f1 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:41.018763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 644/1291] 10.222.0.40 () {56 vars in 1116 bytes} [Fri Jul 29 04:37:40 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:41.025130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ec4b22-21ed-4c56-8c8f-4c82f30841a4 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:41.123663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ec4b22-21ed-4c56-8c8f-4c82f30841a4 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:41.157351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4537df4-4176-4342-8ef9-6f1c28ecea8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:41.157962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4537df4-4176-4342-8ef9-6f1c28ecea8b None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:41.158251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4537df4-4176-4342-8ef9-6f1c28ecea8b None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:41.158533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4537df4-4176-4342-8ef9-6f1c28ecea8b None admin] PATH_INFO: `/v3/services` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:41.159217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4537df4-4176-4342-8ef9-6f1c28ecea8b None admin] RBAC: Adding query filter params (type=placement) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:41.159526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4537df4-4176-4342-8ef9-6f1c28ecea8b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:41.160938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4537df4-4176-4342-8ef9-6f1c28ecea8b None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:41.165511 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 648/1292] 10.222.0.40 () {56 vars in 1116 bytes} [Fri Jul 29 04:37:41 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:43.175602 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cfb07b-4280-4877-9868-ca0ab4ebd2ab None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:43.176151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cfb07b-4280-4877-9868-ca0ab4ebd2ab None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:43.176151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cfb07b-4280-4877-9868-ca0ab4ebd2ab None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:43.177571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 645/1293] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:37:43.183096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad81bff-5440-44a0-83c7-a31f9887b038 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:43.183546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad81bff-5440-44a0-83c7-a31f9887b038 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:43.183648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad81bff-5440-44a0-83c7-a31f9887b038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:43.202251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-2ad81bff-5440-44a0-83c7-a31f9887b038 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:43.203446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2ad81bff-5440-44a0-83c7-a31f9887b038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:43.232442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:43.276946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 649/1294] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:43 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:37:43.305565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cfb07b-4280-4877-9868-ca0ab4ebd2ab None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:43.340831 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:43.410547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cfb07b-4280-4877-9868-ca0ab4ebd2ab None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:43.444052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c23d85b-8584-4c41-8960-59ff087afaa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:43.444718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c23d85b-8584-4c41-8960-59ff087afaa2 None admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:43.445006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c23d85b-8584-4c41-8960-59ff087afaa2 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:43.445286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c23d85b-8584-4c41-8960-59ff087afaa2 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:43.445991 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c23d85b-8584-4c41-8960-59ff087afaa2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:43.447348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c23d85b-8584-4c41-8960-59ff087afaa2 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:43.461356 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 646/1295] 10.222.0.40 () {60 vars in 1141 bytes} [Fri Jul 29 04:37:43 2022] POST /identity/v3/services => generated 246 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:37:45.447539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71d75c9-f21d-46c5-8ed7-fa1c5dc3cdae None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:45.448069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71d75c9-f21d-46c5-8ed7-fa1c5dc3cdae None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:45.448192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71d75c9-f21d-46c5-8ed7-fa1c5dc3cdae None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:45.449503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 650/1296] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:37:45.454687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fd6e74-b675-4155-89df-f4a02dddbdfc None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:45.455075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fd6e74-b675-4155-89df-f4a02dddbdfc None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:45.455254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fd6e74-b675-4155-89df-f4a02dddbdfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:45.473014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-09fd6e74-b675-4155-89df-f4a02dddbdfc None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:45.474218 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-09fd6e74-b675-4155-89df-f4a02dddbdfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:45.545874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 647/1297] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:45 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 29 04:37:45.568708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71d75c9-f21d-46c5-8ed7-fa1c5dc3cdae None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:45.669951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71d75c9-f21d-46c5-8ed7-fa1c5dc3cdae None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:45.704207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e158ea-db37-49f5-a37d-47b03d92f4df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:45.704879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e158ea-db37-49f5-a37d-47b03d92f4df None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:45.705180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e158ea-db37-49f5-a37d-47b03d92f4df None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:45.705444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e158ea-db37-49f5-a37d-47b03d92f4df None admin] PATH_INFO: `/v3/services/placement` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:45.706068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e158ea-db37-49f5-a37d-47b03d92f4df None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:45.707448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e158ea-db37-49f5-a37d-47b03d92f4df None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-95e158ea-db37-49f5-a37d-47b03d92f4df None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:45.716757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:45.718078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:45.719318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 651/1298] 10.222.0.40 () {56 vars in 1117 bytes} [Fri Jul 29 04:37:45 2022] GET /identity/v3/services/placement => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:45.723919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fd6e74-b675-4155-89df-f4a02dddbdfc None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:45.742468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:45.820584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fd6e74-b675-4155-89df-f4a02dddbdfc None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:45.854058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09e0b80-7d7d-4188-a12f-5e3e38a3556b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:45.854671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09e0b80-7d7d-4188-a12f-5e3e38a3556b None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:45.854951 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09e0b80-7d7d-4188-a12f-5e3e38a3556b None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:45.855223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09e0b80-7d7d-4188-a12f-5e3e38a3556b None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:45.855895 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09e0b80-7d7d-4188-a12f-5e3e38a3556b None admin] RBAC: Adding query filter params (name=placement) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:45.856200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09e0b80-7d7d-4188-a12f-5e3e38a3556b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:45.857576 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09e0b80-7d7d-4188-a12f-5e3e38a3556b None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:45.862524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 648/1299] 10.222.0.40 () {56 vars in 1116 bytes} [Fri Jul 29 04:37:45 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:45.868440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e158ea-db37-49f5-a37d-47b03d92f4df None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:45.899971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:45.966386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e158ea-db37-49f5-a37d-47b03d92f4df None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:45.998941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a0a3af-9f35-4f9a-bd58-216dfc8904f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:45.999555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a0a3af-9f35-4f9a-bd58-216dfc8904f4 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:45.999835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a0a3af-9f35-4f9a-bd58-216dfc8904f4 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:46.000107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a0a3af-9f35-4f9a-bd58-216dfc8904f4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:46.000884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a0a3af-9f35-4f9a-bd58-216dfc8904f4 None admin] RBAC: Adding query filter params (interface=public, service_id=c08a43e95784410981b963db581d896d, region_id=RegionOne) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:46.001197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a0a3af-9f35-4f9a-bd58-216dfc8904f4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:46.002563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a0a3af-9f35-4f9a-bd58-216dfc8904f4 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:46.007489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 652/1300] 10.222.0.40 () {56 vars in 1251 bytes} [Fri Jul 29 04:37:45 2022] GET /identity/v3/endpoints?service_id=c08a43e95784410981b963db581d896d&interface=public®ion_id=RegionOne => generated 194 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:46.013064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09e0b80-7d7d-4188-a12f-5e3e38a3556b None admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:46.091539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:46.109584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09e0b80-7d7d-4188-a12f-5e3e38a3556b None admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:46.142626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32779f0e-818d-40b9-9914-4102e36ec718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:46.143238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32779f0e-818d-40b9-9914-4102e36ec718 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:46.143524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32779f0e-818d-40b9-9914-4102e36ec718 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:46.143802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32779f0e-818d-40b9-9914-4102e36ec718 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:46.144454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32779f0e-818d-40b9-9914-4102e36ec718 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:46.145844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32779f0e-818d-40b9-9914-4102e36ec718 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:46.153197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 649/1301] 10.222.0.40 () {56 vars in 1087 bytes} [Fri Jul 29 04:37:46 2022] GET /identity/v3/services => generated 2191 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:37:48.133998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6def61c7-ec54-4579-b1b6-900880db0baf None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:48.134559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6def61c7-ec54-4579-b1b6-900880db0baf None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:48.134559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6def61c7-ec54-4579-b1b6-900880db0baf None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:48.135907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 653/1302] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:37:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:37:48.140969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6268e5-da51-45e2-8cf1-5b6f8bb6c47c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:48.141375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6268e5-da51-45e2-8cf1-5b6f8bb6c47c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:48.141541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6268e5-da51-45e2-8cf1-5b6f8bb6c47c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:48.159778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9f6268e5-da51-45e2-8cf1-5b6f8bb6c47c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:37:48.161016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9f6268e5-da51-45e2-8cf1-5b6f8bb6c47c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:37:48.226961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 650/1303] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:37:48 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 29 04:37:48.248993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6def61c7-ec54-4579-b1b6-900880db0baf None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:48.347305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:48.354548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6def61c7-ec54-4579-b1b6-900880db0baf None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:48.387428 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c8b04e-0885-4bb7-943b-697daea77ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:48.388049 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c8b04e-0885-4bb7-943b-697daea77ea5 None admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:48.388335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c8b04e-0885-4bb7-943b-697daea77ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:48.388636 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c8b04e-0885-4bb7-943b-697daea77ea5 None admin] PATH_INFO: `/v3/services/placement` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:48.389271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c8b04e-0885-4bb7-943b-697daea77ea5 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:48.390672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c8b04e-0885-4bb7-943b-697daea77ea5 None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application [None req-a9c8b04e-0885-4bb7-943b-697daea77ea5 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1517, in full_dispatch_request Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1503, in dispatch_request Jul 29 04:37:48.400000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 29 04:37:48.401202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.get_or_create( Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application with Lock( Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self._enter() Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application return self.creator() Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application created_value = creator( Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: ERROR keystone.server.flask.application Jul 29 04:37:48.402211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 654/1304] 10.222.0.40 () {56 vars in 1117 bytes} [Fri Jul 29 04:37:48 2022] GET /identity/v3/services/placement => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 29 04:37:48.407375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6268e5-da51-45e2-8cf1-5b6f8bb6c47c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:48.481782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:48.504918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6268e5-da51-45e2-8cf1-5b6f8bb6c47c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:48.538675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2af4b2-d6d5-4462-9732-fd7d420beaf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:48.539292 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2af4b2-d6d5-4462-9732-fd7d420beaf3 None admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:48.539582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2af4b2-d6d5-4462-9732-fd7d420beaf3 None admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:48.539862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2af4b2-d6d5-4462-9732-fd7d420beaf3 None admin] PATH_INFO: `/v3/services` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:48.540546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2af4b2-d6d5-4462-9732-fd7d420beaf3 None admin] RBAC: Adding query filter params (name=placement) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:37:48.540888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2af4b2-d6d5-4462-9732-fd7d420beaf3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:48.542256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2af4b2-d6d5-4462-9732-fd7d420beaf3 None admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:48.547542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 651/1305] 10.222.0.40 () {56 vars in 1116 bytes} [Fri Jul 29 04:37:48 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:37:48.553298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c8b04e-0885-4bb7-943b-697daea77ea5 None admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:37:48.652236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c8b04e-0885-4bb7-943b-697daea77ea5 None admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:37:48.688129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d891d61-24bd-486d-ab5a-9ba2415978ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:37:48.688776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d891d61-24bd-486d-ab5a-9ba2415978ac None admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:37:48.689069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d891d61-24bd-486d-ab5a-9ba2415978ac None admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:37:48.689351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d891d61-24bd-486d-ab5a-9ba2415978ac None admin] PATH_INFO: `/v3/endpoints` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:37:48.690070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d891d61-24bd-486d-ab5a-9ba2415978ac None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:37:48.691463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d891d61-24bd-486d-ab5a-9ba2415978ac None admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:37:48.710951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 655/1306] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:37:48 2022] POST /identity/v3/endpoints => generated 331 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:37:58.544293 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=30 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:37:58.706651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:38:51.453887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83b921d-f0e3-41c2-986e-0bda7a2303b5 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:51.454462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83b921d-f0e3-41c2-986e-0bda7a2303b5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:51.454462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83b921d-f0e3-41c2-986e-0bda7a2303b5 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:51.459566 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 652/1307] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:38:51 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:38:51.461022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738aaa80-4af0-4228-943a-62eb70850a0e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:51.461466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738aaa80-4af0-4228-943a-62eb70850a0e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:51.461562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738aaa80-4af0-4228-943a-62eb70850a0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:51.605314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-738aaa80-4af0-4228-943a-62eb70850a0e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:38:51.607078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-738aaa80-4af0-4228-943a-62eb70850a0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:38:51.818922 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 656/1308] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:38:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 359 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:38:51.863901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f35806e-c03c-4d1d-b2dc-6e709083bfa0 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:51.864188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f35806e-c03c-4d1d-b2dc-6e709083bfa0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:51.864659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f35806e-c03c-4d1d-b2dc-6e709083bfa0 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:51.866048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 653/1309] 10.222.0.40 () {54 vars in 965 bytes} [Fri Jul 29 04:38:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:38:51.872251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baf40ef-221b-42a8-b4b7-22ed59f55df7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:51.872704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baf40ef-221b-42a8-b4b7-22ed59f55df7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:51.873101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5baf40ef-221b-42a8-b4b7-22ed59f55df7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:51.953437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-5baf40ef-221b-42a8-b4b7-22ed59f55df7 None None] MFA Rules not processed for user `f6726ef0c4184040a18a76898bb738d1`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:38:51.954885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5baf40ef-221b-42a8-b4b7-22ed59f55df7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:38:51.977019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:38:52.038648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 657/1310] 10.222.0.40 () {58 vars in 1062 bytes} [Fri Jul 29 04:38:51 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:38:52.047939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f35806e-c03c-4d1d-b2dc-6e709083bfa0 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:38:52.050401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7f35806e-c03c-4d1d-b2dc-6e709083bfa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:38:52.177687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f35806e-c03c-4d1d-b2dc-6e709083bfa0 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:38:52.261741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f660814d-e7ff-4bee-bfad-2bf2ca2151e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:38:52.262421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f660814d-e7ff-4bee-bfad-2bf2ca2151e8 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:52.262701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f660814d-e7ff-4bee-bfad-2bf2ca2151e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:52.262976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f660814d-e7ff-4bee-bfad-2bf2ca2151e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:52.264693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f660814d-e7ff-4bee-bfad-2bf2ca2151e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:38:52.327618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:38:52.389172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f660814d-e7ff-4bee-bfad-2bf2ca2151e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:38:52.390732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f660814d-e7ff-4bee-bfad-2bf2ca2151e8 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:38:52.411967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 654/1311] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:38:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:38:54.722345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445a1719-8349-4898-ba4a-877d2bbdd617 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:54.722921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445a1719-8349-4898-ba4a-877d2bbdd617 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:54.722921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445a1719-8349-4898-ba4a-877d2bbdd617 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:54.724496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 658/1312] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:38:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:38:54.730277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f712f74-2e1d-4f37-b8b2-a7475df235c3 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:54.731143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f712f74-2e1d-4f37-b8b2-a7475df235c3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:54.731428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f712f74-2e1d-4f37-b8b2-a7475df235c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:54.828021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-4f712f74-2e1d-4f37-b8b2-a7475df235c3 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:38:54.829309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-4f712f74-2e1d-4f37-b8b2-a7475df235c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:38:54.940641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 655/1313] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:38:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:38:54.979365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445a1719-8349-4898-ba4a-877d2bbdd617 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:38:54.980970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-445a1719-8349-4898-ba4a-877d2bbdd617 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:38:55.101874 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445a1719-8349-4898-ba4a-877d2bbdd617 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:38:55.178377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:38:55.181195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a160f7b-b8ba-4733-bf6d-452003377930 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:38:55.182035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a160f7b-b8ba-4733-bf6d-452003377930 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:55.182497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a160f7b-b8ba-4733-bf6d-452003377930 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:55.182786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a160f7b-b8ba-4733-bf6d-452003377930 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:55.184713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3a160f7b-b8ba-4733-bf6d-452003377930 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:38:55.295913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a160f7b-b8ba-4733-bf6d-452003377930 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:38:55.297649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a160f7b-b8ba-4733-bf6d-452003377930 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:38:55.317753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 659/1314] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:38:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:38:57.655366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcff5d5-923f-47cd-b1b0-22453fcddf91 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:57.655834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcff5d5-923f-47cd-b1b0-22453fcddf91 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:57.655947 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcff5d5-923f-47cd-b1b0-22453fcddf91 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:57.657374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 656/1315] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:38:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:38:57.663273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efba908f-f7b9-4960-99a3-eecc1a3e3156 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:57.663742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efba908f-f7b9-4960-99a3-eecc1a3e3156 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:57.663831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efba908f-f7b9-4960-99a3-eecc1a3e3156 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:57.732422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-efba908f-f7b9-4960-99a3-eecc1a3e3156 None None] MFA Rules not processed for user `0c530b748ab741c89390231f1b4f34c9`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:38:57.733661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-efba908f-f7b9-4960-99a3-eecc1a3e3156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:38:57.786520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:38:57.832490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 660/1316] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:38:57 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:38:57.872522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcff5d5-923f-47cd-b1b0-22453fcddf91 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:38:57.874116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9fcff5d5-923f-47cd-b1b0-22453fcddf91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:38:57.884107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:38:57.978658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcff5d5-923f-47cd-b1b0-22453fcddf91 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:38:58.050550 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea78445-3562-44ed-ac8d-11db4f65bcda service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:38:58.051178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea78445-3562-44ed-ac8d-11db4f65bcda service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:38:58.051466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea78445-3562-44ed-ac8d-11db4f65bcda service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:38:58.051751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea78445-3562-44ed-ac8d-11db4f65bcda service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:38:58.053427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-6ea78445-3562-44ed-ac8d-11db4f65bcda service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:38:58.116343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:38:58.164213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea78445-3562-44ed-ac8d-11db4f65bcda service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:38:58.165679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea78445-3562-44ed-ac8d-11db4f65bcda service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:38:58.184164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 657/1317] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:38:57 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:00.709605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db79588c-97e6-45bd-a026-8ac07ae8ad05 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:00.710134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db79588c-97e6-45bd-a026-8ac07ae8ad05 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:00.710231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db79588c-97e6-45bd-a026-8ac07ae8ad05 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:00.711548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 661/1318] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:00.716784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d5f83e-fb69-4b7b-b97d-70fc353527a2 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:00.717209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d5f83e-fb69-4b7b-b97d-70fc353527a2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:00.717358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d5f83e-fb69-4b7b-b97d-70fc353527a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:00.782945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-08d5f83e-fb69-4b7b-b97d-70fc353527a2 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:00.784821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-08d5f83e-fb69-4b7b-b97d-70fc353527a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:00.878831 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 658/1319] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:00 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:00.917989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db79588c-97e6-45bd-a026-8ac07ae8ad05 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:00.919814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-db79588c-97e6-45bd-a026-8ac07ae8ad05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:01.022370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db79588c-97e6-45bd-a026-8ac07ae8ad05 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:01.095419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b665e1e1-0c09-4d3f-9946-301905ba5788 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:01.096151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665e1e1-0c09-4d3f-9946-301905ba5788 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:01.096435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665e1e1-0c09-4d3f-9946-301905ba5788 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:01.096744 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b665e1e1-0c09-4d3f-9946-301905ba5788 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:01.098376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b665e1e1-0c09-4d3f-9946-301905ba5788 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:01.105627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:01.207067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b665e1e1-0c09-4d3f-9946-301905ba5788 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:01.208462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b665e1e1-0c09-4d3f-9946-301905ba5788 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:01.226265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 662/1320] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:00 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:03.906084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f2802-4512-42ff-bbf3-bd5cf3eb9216 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:03.906616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f2802-4512-42ff-bbf3-bd5cf3eb9216 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:03.906616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733f2802-4512-42ff-bbf3-bd5cf3eb9216 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:03.908005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 659/1321] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:03.914245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d845c66-169c-4e69-859c-070886c2d4bf None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:03.914659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d845c66-169c-4e69-859c-070886c2d4bf None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:03.915001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d845c66-169c-4e69-859c-070886c2d4bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:03.981612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-5d845c66-169c-4e69-859c-070886c2d4bf None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:03.982848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5d845c66-169c-4e69-859c-070886c2d4bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:04.043832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:04.092101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 663/1322] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:04.129584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733f2802-4512-42ff-bbf3-bd5cf3eb9216 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:04.131231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-733f2802-4512-42ff-bbf3-bd5cf3eb9216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:04.152354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:04.244361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733f2802-4512-42ff-bbf3-bd5cf3eb9216 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:04.322327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a727d2-c347-4ab3-b493-746a5fc11957 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:04.322960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a727d2-c347-4ab3-b493-746a5fc11957 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:04.323251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a727d2-c347-4ab3-b493-746a5fc11957 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:04.323535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a727d2-c347-4ab3-b493-746a5fc11957 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:04.325219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-50a727d2-c347-4ab3-b493-746a5fc11957 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:04.397923 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:04.438653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a727d2-c347-4ab3-b493-746a5fc11957 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:04.440757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a727d2-c347-4ab3-b493-746a5fc11957 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:04.461176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 660/1323] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:06.535545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c0233-c614-45a6-9852-d6c04784ce52 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:06.536070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c0233-c614-45a6-9852-d6c04784ce52 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:06.536070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c0233-c614-45a6-9852-d6c04784ce52 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:06.537405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 664/1324] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:06.542596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84969ec9-bdf8-4b8e-9a41-05f764d8d13d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:06.543003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84969ec9-bdf8-4b8e-9a41-05f764d8d13d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:06.543146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84969ec9-bdf8-4b8e-9a41-05f764d8d13d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:06.608487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-84969ec9-bdf8-4b8e-9a41-05f764d8d13d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:06.609736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-84969ec9-bdf8-4b8e-9a41-05f764d8d13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:06.713434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 661/1325] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:06.749980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228c0233-c614-45a6-9852-d6c04784ce52 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:06.751544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-228c0233-c614-45a6-9852-d6c04784ce52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:06.858331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228c0233-c614-45a6-9852-d6c04784ce52 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:06.937964 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dc5ff4-9557-4d23-be90-754cf6f01653 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:06.938954 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dc5ff4-9557-4d23-be90-754cf6f01653 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:06.939498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dc5ff4-9557-4d23-be90-754cf6f01653 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:06.940110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dc5ff4-9557-4d23-be90-754cf6f01653 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:06.941403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-78dc5ff4-9557-4d23-be90-754cf6f01653 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:06.958704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:07.057156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dc5ff4-9557-4d23-be90-754cf6f01653 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:07.058610 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dc5ff4-9557-4d23-be90-754cf6f01653 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:07.078204 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 665/1326] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:09.252023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cd0857-c70b-4cba-ab5d-e61f10c2e24f None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:09.252587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cd0857-c70b-4cba-ab5d-e61f10c2e24f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:09.252587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cd0857-c70b-4cba-ab5d-e61f10c2e24f None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:09.253956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 662/1327] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:09.259081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8adce0f-9f30-4083-93da-7a347af16083 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:09.259537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8adce0f-9f30-4083-93da-7a347af16083 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:09.259647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8adce0f-9f30-4083-93da-7a347af16083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:09.329118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-a8adce0f-9f30-4083-93da-7a347af16083 None None] MFA Rules not processed for user `0c530b748ab741c89390231f1b4f34c9`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:09.330884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a8adce0f-9f30-4083-93da-7a347af16083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:09.398013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:09.433502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 666/1328] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:09 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:09.471524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cd0857-c70b-4cba-ab5d-e61f10c2e24f None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:09.473222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f5cd0857-c70b-4cba-ab5d-e61f10c2e24f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:09.505007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:09.579352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cd0857-c70b-4cba-ab5d-e61f10c2e24f None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:09.654631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afc2731-e2a3-4e70-885d-edcc26e8e737 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:09.655269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afc2731-e2a3-4e70-885d-edcc26e8e737 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:09.655556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afc2731-e2a3-4e70-885d-edcc26e8e737 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:09.655829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afc2731-e2a3-4e70-885d-edcc26e8e737 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:09.657497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8afc2731-e2a3-4e70-885d-edcc26e8e737 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:09.736513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:09.770484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afc2731-e2a3-4e70-885d-edcc26e8e737 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:09.771938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afc2731-e2a3-4e70-885d-edcc26e8e737 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:09.792124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 663/1329] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:09 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:12.076143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3658386d-d794-467d-91b1-84dcae52b418 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:12.076729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3658386d-d794-467d-91b1-84dcae52b418 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:12.076729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3658386d-d794-467d-91b1-84dcae52b418 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:12.078079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 667/1330] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:12.083862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611ffaef-0ebc-4bf3-bf03-d74a1ca06bd5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:12.084265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611ffaef-0ebc-4bf3-bf03-d74a1ca06bd5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:12.084436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611ffaef-0ebc-4bf3-bf03-d74a1ca06bd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:12.150012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-611ffaef-0ebc-4bf3-bf03-d74a1ca06bd5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:12.151257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-611ffaef-0ebc-4bf3-bf03-d74a1ca06bd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:12.248717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 664/1331] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:12.301920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7482aab9-c420-4885-984c-5725b002ed17 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:12.302211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7482aab9-c420-4885-984c-5725b002ed17 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:12.302493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7482aab9-c420-4885-984c-5725b002ed17 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:12.303892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 668/1332] 10.222.0.40 () {54 vars in 965 bytes} [Fri Jul 29 04:39:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:12.309666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4687757c-b048-4016-9d1d-2f79c5c39de1 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:12.309960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4687757c-b048-4016-9d1d-2f79c5c39de1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:12.310239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4687757c-b048-4016-9d1d-2f79c5c39de1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:12.355653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:12.386491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-4687757c-b048-4016-9d1d-2f79c5c39de1 None None] MFA Rules not processed for user `f6726ef0c4184040a18a76898bb738d1`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:12.387709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-4687757c-b048-4016-9d1d-2f79c5c39de1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:12.468938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 665/1333] 10.222.0.40 () {58 vars in 1062 bytes} [Fri Jul 29 04:39:12 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:12.478203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7482aab9-c420-4885-984c-5725b002ed17 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:12.479724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7482aab9-c420-4885-984c-5725b002ed17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:12.582897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7482aab9-c420-4885-984c-5725b002ed17 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:12.654973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66989483-1bb5-4536-b1ae-a85b52e128b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:12.655594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66989483-1bb5-4536-b1ae-a85b52e128b6 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:12.655879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66989483-1bb5-4536-b1ae-a85b52e128b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:12.656157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66989483-1bb5-4536-b1ae-a85b52e128b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:12.657817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-66989483-1bb5-4536-b1ae-a85b52e128b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:12.684966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:12.765724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66989483-1bb5-4536-b1ae-a85b52e128b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:12.767237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66989483-1bb5-4536-b1ae-a85b52e128b6 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:12.784907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 669/1334] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:15.054784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91620853-9a37-43da-86c6-78d24efeb822 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:15.055327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91620853-9a37-43da-86c6-78d24efeb822 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:15.055327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91620853-9a37-43da-86c6-78d24efeb822 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:15.056715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 666/1335] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:15.061859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2850e048-e1d2-4500-ad60-5742205372ba None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:15.062268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2850e048-e1d2-4500-ad60-5742205372ba None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:15.062425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2850e048-e1d2-4500-ad60-5742205372ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:15.128222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-2850e048-e1d2-4500-ad60-5742205372ba None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:15.129477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2850e048-e1d2-4500-ad60-5742205372ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:15.203606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:15.226867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 670/1336] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:15.267464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91620853-9a37-43da-86c6-78d24efeb822 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:15.269036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-91620853-9a37-43da-86c6-78d24efeb822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:15.370305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91620853-9a37-43da-86c6-78d24efeb822 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:15.397402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:15.442252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a187d4-5159-4858-aa1b-0e52f03becca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:15.442864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a187d4-5159-4858-aa1b-0e52f03becca service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:15.443146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a187d4-5159-4858-aa1b-0e52f03becca service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:15.443419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a187d4-5159-4858-aa1b-0e52f03becca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:15.445049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-22a187d4-5159-4858-aa1b-0e52f03becca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:15.551786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a187d4-5159-4858-aa1b-0e52f03becca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:15.553172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a187d4-5159-4858-aa1b-0e52f03becca service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:15.570673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 667/1337] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:18.224419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3cd803-b9ab-413b-a647-45f05f7cfe18 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:18.225029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3cd803-b9ab-413b-a647-45f05f7cfe18 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:18.225029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3cd803-b9ab-413b-a647-45f05f7cfe18 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:18.226369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 671/1338] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:18.231696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583c7a33-8882-43d2-b255-517b5efe55ca None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:18.232126 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583c7a33-8882-43d2-b255-517b5efe55ca None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:18.232257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583c7a33-8882-43d2-b255-517b5efe55ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:18.299044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-583c7a33-8882-43d2-b255-517b5efe55ca None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:18.300257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-583c7a33-8882-43d2-b255-517b5efe55ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:18.308361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:18.401997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 668/1339] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:18 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:18.440364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3cd803-b9ab-413b-a647-45f05f7cfe18 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:18.441959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ae3cd803-b9ab-413b-a647-45f05f7cfe18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:18.551956 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3cd803-b9ab-413b-a647-45f05f7cfe18 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:18.623866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6de1a3-dd37-49c9-bf6a-8a33639006ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:18.624480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6de1a3-dd37-49c9-bf6a-8a33639006ba service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:18.624795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6de1a3-dd37-49c9-bf6a-8a33639006ba service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:18.625070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6de1a3-dd37-49c9-bf6a-8a33639006ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:18.626711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2d6de1a3-dd37-49c9-bf6a-8a33639006ba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:18.662351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:18.736699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6de1a3-dd37-49c9-bf6a-8a33639006ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:18.738146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6de1a3-dd37-49c9-bf6a-8a33639006ba service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:18.755854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 672/1340] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:18 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:22.842023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af1879-4ce9-4e59-a01f-b7a3224af3c2 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:22.842527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af1879-4ce9-4e59-a01f-b7a3224af3c2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:22.842527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07af1879-4ce9-4e59-a01f-b7a3224af3c2 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:22.843882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 669/1341] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:22.849641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed561a3-899b-41c3-a51f-0fed44382f5f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:22.850078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed561a3-899b-41c3-a51f-0fed44382f5f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:22.850237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed561a3-899b-41c3-a51f-0fed44382f5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:22.918333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-3ed561a3-899b-41c3-a51f-0fed44382f5f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:22.919559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3ed561a3-899b-41c3-a51f-0fed44382f5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:23.005242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:23.017861 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 673/1342] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:23.056283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07af1879-4ce9-4e59-a01f-b7a3224af3c2 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:23.057962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-07af1879-4ce9-4e59-a01f-b7a3224af3c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:23.165004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07af1879-4ce9-4e59-a01f-b7a3224af3c2 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:23.202558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:23.240557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a1abaf-e149-480b-b0ff-9f0be6c46eb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:23.241217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a1abaf-e149-480b-b0ff-9f0be6c46eb3 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:23.241505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a1abaf-e149-480b-b0ff-9f0be6c46eb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:23.241784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a1abaf-e149-480b-b0ff-9f0be6c46eb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:23.243423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-46a1abaf-e149-480b-b0ff-9f0be6c46eb3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:23.361848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a1abaf-e149-480b-b0ff-9f0be6c46eb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:23.363193 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a1abaf-e149-480b-b0ff-9f0be6c46eb3 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:23.381374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 670/1343] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:25.990356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab014221-23b4-4a22-bf9f-49f0cfb53ae8 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:25.990940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab014221-23b4-4a22-bf9f-49f0cfb53ae8 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:25.990940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab014221-23b4-4a22-bf9f-49f0cfb53ae8 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:25.992268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 674/1344] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:25.998101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155f2862-8817-4021-8117-e2544d2807a3 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:25.998635 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155f2862-8817-4021-8117-e2544d2807a3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:25.998727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155f2862-8817-4021-8117-e2544d2807a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:26.072900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-155f2862-8817-4021-8117-e2544d2807a3 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:26.074151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-155f2862-8817-4021-8117-e2544d2807a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:26.093767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:26.175465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 671/1345] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:25 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:26.213187 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab014221-23b4-4a22-bf9f-49f0cfb53ae8 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:26.214735 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ab014221-23b4-4a22-bf9f-49f0cfb53ae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:26.329934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab014221-23b4-4a22-bf9f-49f0cfb53ae8 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:26.403197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3583c2b3-4275-4814-b0ce-dc334b71b9f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:26.403736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3583c2b3-4275-4814-b0ce-dc334b71b9f7 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:26.404023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3583c2b3-4275-4814-b0ce-dc334b71b9f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:26.404302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3583c2b3-4275-4814-b0ce-dc334b71b9f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:26.405960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3583c2b3-4275-4814-b0ce-dc334b71b9f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:26.450641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:26.518284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3583c2b3-4275-4814-b0ce-dc334b71b9f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:26.519728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3583c2b3-4275-4814-b0ce-dc334b71b9f7 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:26.538214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 675/1346] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:26 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:30.503225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff53a4-f402-4d79-9c71-051e9e219149 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:30.503774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff53a4-f402-4d79-9c71-051e9e219149 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:30.503959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff53a4-f402-4d79-9c71-051e9e219149 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:30.505512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 672/1347] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:30.510816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d79d11-e367-406c-8b96-7f086441323b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:30.511224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d79d11-e367-406c-8b96-7f086441323b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:30.511531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d79d11-e367-406c-8b96-7f086441323b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:30.578344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-77d79d11-e367-406c-8b96-7f086441323b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:30.579559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-77d79d11-e367-406c-8b96-7f086441323b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:30.681011 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:30.683732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 676/1348] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:30.722112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff53a4-f402-4d79-9c71-051e9e219149 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:30.723752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-47ff53a4-f402-4d79-9c71-051e9e219149 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:30.847656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff53a4-f402-4d79-9c71-051e9e219149 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:30.899892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:30.927849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f58b4d1-9200-4836-a8fc-dc960d2b4058 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:30.928486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f58b4d1-9200-4836-a8fc-dc960d2b4058 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:30.928804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f58b4d1-9200-4836-a8fc-dc960d2b4058 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:30.929089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f58b4d1-9200-4836-a8fc-dc960d2b4058 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:30.930826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2f58b4d1-9200-4836-a8fc-dc960d2b4058 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:31.055190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f58b4d1-9200-4836-a8fc-dc960d2b4058 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:31.056797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f58b4d1-9200-4836-a8fc-dc960d2b4058 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:31.078020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 673/1349] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:35.961169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1314e38-de8a-4201-ac07-ea5f31906528 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:35.961592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1314e38-de8a-4201-ac07-ea5f31906528 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:35.964202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1314e38-de8a-4201-ac07-ea5f31906528 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:35.964202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 677/1350] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:39:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:35.969414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0f4e0-4231-489b-abd1-972ca0c55138 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:35.969884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0f4e0-4231-489b-abd1-972ca0c55138 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:35.970033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0f4e0-4231-489b-abd1-972ca0c55138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:36.107933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-b5f0f4e0-4231-489b-abd1-972ca0c55138 None None] MFA Rules not processed for user `e3da5e428049419a9b35494655f52484`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:36.110816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b5f0f4e0-4231-489b-abd1-972ca0c55138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:36.145607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:36.243097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 674/1351] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:39:35 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 275 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:39:40.393217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda709be-c9c9-45d4-a0a5-1605d87a3251 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:40.394088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda709be-c9c9-45d4-a0a5-1605d87a3251 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:40.394367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda709be-c9c9-45d4-a0a5-1605d87a3251 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:40.396042 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 678/1352] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:39:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:40.401430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462b3b1-8ba8-4dab-be7a-c3f1758c22ca None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:40.402268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462b3b1-8ba8-4dab-be7a-c3f1758c22ca None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:40.402729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462b3b1-8ba8-4dab-be7a-c3f1758c22ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:40.486922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9462b3b1-8ba8-4dab-be7a-c3f1758c22ca None None] MFA Rules not processed for user `e3da5e428049419a9b35494655f52484`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:40.488438 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9462b3b1-8ba8-4dab-be7a-c3f1758c22ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:40.565033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:40.604166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 675/1353] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:39:40 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:41.930924 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38767885-9ce4-46ce-97f1-3e5a6f8b2653 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:41.931432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38767885-9ce4-46ce-97f1-3e5a6f8b2653 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:41.931432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38767885-9ce4-46ce-97f1-3e5a6f8b2653 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:41.932896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 679/1354] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:39:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:41.938425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b212fb-d611-4795-bcc5-93a00f928f4d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:41.938853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b212fb-d611-4795-bcc5-93a00f928f4d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:41.939014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b212fb-d611-4795-bcc5-93a00f928f4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:42.016905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-64b212fb-d611-4795-bcc5-93a00f928f4d None None] MFA Rules not processed for user `e3da5e428049419a9b35494655f52484`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:42.018142 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-64b212fb-d611-4795-bcc5-93a00f928f4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:42.126742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 676/1355] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:39:41 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:45.639028 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ee3d16-7291-45ac-9a9b-898ee8ad38af None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:45.640109 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ee3d16-7291-45ac-9a9b-898ee8ad38af None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:45.640698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ee3d16-7291-45ac-9a9b-898ee8ad38af None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:45.644690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 680/1356] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:45 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:45.648884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0f72c1-7b6b-4d45-a5e4-033660621c3e None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:45.649807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0f72c1-7b6b-4d45-a5e4-033660621c3e None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:45.650260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0f72c1-7b6b-4d45-a5e4-033660621c3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:45.706037 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:45.743826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-3f0f72c1-7b6b-4d45-a5e4-033660621c3e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:45.745132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3f0f72c1-7b6b-4d45-a5e4-033660621c3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:45.850244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 677/1357] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:45.903567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda77d07-10cc-422b-b3fd-15087009fac1 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:45.904430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda77d07-10cc-422b-b3fd-15087009fac1 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:45.905098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda77d07-10cc-422b-b3fd-15087009fac1 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:45.907190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 681/1358] 10.222.0.40 () {54 vars in 962 bytes} [Fri Jul 29 04:39:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:45.919905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3549ce1d-24d2-40b7-9b7d-5decae0aea83 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:45.920195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3549ce1d-24d2-40b7-9b7d-5decae0aea83 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:45.920474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3549ce1d-24d2-40b7-9b7d-5decae0aea83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:46.003959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-3549ce1d-24d2-40b7-9b7d-5decae0aea83 None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:46.005223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3549ce1d-24d2-40b7-9b7d-5decae0aea83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:46.056335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:46.130487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 678/1359] 10.222.0.40 () {58 vars in 1059 bytes} [Fri Jul 29 04:39:45 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:46.137990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda77d07-10cc-422b-b3fd-15087009fac1 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:46.140385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-cda77d07-10cc-422b-b3fd-15087009fac1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:46.301835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda77d07-10cc-422b-b3fd-15087009fac1 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:46.383082 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:46.428447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fab4e3-5e64-4f60-a39d-afff1753b7dc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:46.429096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fab4e3-5e64-4f60-a39d-afff1753b7dc service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:46.429379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fab4e3-5e64-4f60-a39d-afff1753b7dc service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:46.429656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fab4e3-5e64-4f60-a39d-afff1753b7dc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:46.431272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b9fab4e3-5e64-4f60-a39d-afff1753b7dc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:46.533365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fab4e3-5e64-4f60-a39d-afff1753b7dc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:46.535019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fab4e3-5e64-4f60-a39d-afff1753b7dc service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:46.554851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 682/1360] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:48.798494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6dd771-cf8a-4701-9b8f-fe50384fc969 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:48.799035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6dd771-cf8a-4701-9b8f-fe50384fc969 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:48.799035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6dd771-cf8a-4701-9b8f-fe50384fc969 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:48.800587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 679/1361] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:48.805959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c939a8-83e5-453c-b50a-533946888318 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:48.806524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c939a8-83e5-453c-b50a-533946888318 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:48.806524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c939a8-83e5-453c-b50a-533946888318 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:48.877733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-95c939a8-83e5-453c-b50a-533946888318 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:48.878949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-95c939a8-83e5-453c-b50a-533946888318 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:48.918893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:48.985847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 683/1362] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:49.031715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6dd771-cf8a-4701-9b8f-fe50384fc969 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:49.033500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9f6dd771-cf8a-4701-9b8f-fe50384fc969 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:49.201854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6dd771-cf8a-4701-9b8f-fe50384fc969 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:49.212685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:49.339005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeca44d1-ccec-4875-96cb-1afd864b5a1c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:49.339651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeca44d1-ccec-4875-96cb-1afd864b5a1c service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:49.339651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeca44d1-ccec-4875-96cb-1afd864b5a1c service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:49.340117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeca44d1-ccec-4875-96cb-1afd864b5a1c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:49.342222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-aeca44d1-ccec-4875-96cb-1afd864b5a1c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:49.441038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeca44d1-ccec-4875-96cb-1afd864b5a1c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:49.442675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeca44d1-ccec-4875-96cb-1afd864b5a1c service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:49.462480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 680/1363] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:51.834084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272f5a47-5e79-4195-927b-5d0d4dc3a28e None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:51.834597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272f5a47-5e79-4195-927b-5d0d4dc3a28e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:51.834706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272f5a47-5e79-4195-927b-5d0d4dc3a28e None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:51.836283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 684/1364] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:51.841659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bbcfe3-f6b3-4ce6-84b5-0275b758fbdb None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:51.842090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bbcfe3-f6b3-4ce6-84b5-0275b758fbdb None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:51.842090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bbcfe3-f6b3-4ce6-84b5-0275b758fbdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:51.859104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:51.920231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-96bbcfe3-f6b3-4ce6-84b5-0275b758fbdb None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:51.923236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-96bbcfe3-f6b3-4ce6-84b5-0275b758fbdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:52.029797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 681/1365] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:52.075838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cff9599-f6df-4ce2-b8e1-00e84deee535 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:52.076203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cff9599-f6df-4ce2-b8e1-00e84deee535 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:52.076574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cff9599-f6df-4ce2-b8e1-00e84deee535 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:52.078868 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 685/1366] 10.222.0.40 () {54 vars in 962 bytes} [Fri Jul 29 04:39:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:52.084467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881ab7bc-1651-4453-a4ac-ca570bcd70f2 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:52.084792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881ab7bc-1651-4453-a4ac-ca570bcd70f2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:52.085073 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881ab7bc-1651-4453-a4ac-ca570bcd70f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:52.166610 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-881ab7bc-1651-4453-a4ac-ca570bcd70f2 None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:52.168108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-881ab7bc-1651-4453-a4ac-ca570bcd70f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:52.207818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:52.297405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 682/1367] 10.222.0.40 () {58 vars in 1059 bytes} [Fri Jul 29 04:39:52 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:52.305147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cff9599-f6df-4ce2-b8e1-00e84deee535 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:52.306671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9cff9599-f6df-4ce2-b8e1-00e84deee535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:52.463902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cff9599-f6df-4ce2-b8e1-00e84deee535 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:52.484990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:52.584489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605d1323-2274-40f9-9ce5-812177b307ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:52.585219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605d1323-2274-40f9-9ce5-812177b307ec service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:52.585506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605d1323-2274-40f9-9ce5-812177b307ec service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:52.585788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605d1323-2274-40f9-9ce5-812177b307ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:52.587432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-605d1323-2274-40f9-9ce5-812177b307ec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:52.691344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605d1323-2274-40f9-9ce5-812177b307ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:52.692992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605d1323-2274-40f9-9ce5-812177b307ec service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:52.711849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 686/1368] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:53.810631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9bebf0-348f-4822-ad0f-ee681c4af2d7 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:53.811168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9bebf0-348f-4822-ad0f-ee681c4af2d7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:53.811168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9bebf0-348f-4822-ad0f-ee681c4af2d7 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:53.813265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 683/1369] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:53.820661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47929822-ab52-4df1-9c12-896656191a2f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:53.821165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47929822-ab52-4df1-9c12-896656191a2f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:53.821429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47929822-ab52-4df1-9c12-896656191a2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:53.859912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:53.902356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-47929822-ab52-4df1-9c12-896656191a2f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:53.903849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-47929822-ab52-4df1-9c12-896656191a2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:54.008646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 687/1370] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:54.362678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e450f8a8-503a-441f-afd3-58cd17387d82 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:54.362678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e450f8a8-503a-441f-afd3-58cd17387d82 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:54.362678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e450f8a8-503a-441f-afd3-58cd17387d82 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:54.364257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 684/1371] 10.222.0.40 () {54 vars in 964 bytes} [Fri Jul 29 04:39:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:54.369334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aea6089-2fba-4db2-acc8-a690f050d24f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:54.369742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aea6089-2fba-4db2-acc8-a690f050d24f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:54.370164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aea6089-2fba-4db2-acc8-a690f050d24f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:54.457078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-8aea6089-2fba-4db2-acc8-a690f050d24f None None] MFA Rules not processed for user `de49172e2a054096b319a7e4349ecaea`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:54.458454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8aea6089-2fba-4db2-acc8-a690f050d24f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:54.509111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:54.543209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 688/1372] 10.222.0.40 () {58 vars in 1061 bytes} [Fri Jul 29 04:39:54 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:54.551563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e450f8a8-503a-441f-afd3-58cd17387d82 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:54.553346 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e450f8a8-503a-441f-afd3-58cd17387d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:54.678085 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e450f8a8-503a-441f-afd3-58cd17387d82 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:54.722787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:54.768384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626e1ee4-c604-47cd-a9b1-c2af913b7655 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:54.769049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626e1ee4-c604-47cd-a9b1-c2af913b7655 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:54.769348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626e1ee4-c604-47cd-a9b1-c2af913b7655 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:54.769616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626e1ee4-c604-47cd-a9b1-c2af913b7655 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:54.771247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-626e1ee4-c604-47cd-a9b1-c2af913b7655 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:54.787664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18713293-10da-491c-8948-1a5e8363abbe None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:54.787958 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18713293-10da-491c-8948-1a5e8363abbe None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:54.788229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18713293-10da-491c-8948-1a5e8363abbe None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:54.789911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 689/1373] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:39:54.794956 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52223b64-2bf2-4914-b306-eba3ce2657c3 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:54.795372 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52223b64-2bf2-4914-b306-eba3ce2657c3 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:54.795531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52223b64-2bf2-4914-b306-eba3ce2657c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:54.872570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-52223b64-2bf2-4914-b306-eba3ce2657c3 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:54.875170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-52223b64-2bf2-4914-b306-eba3ce2657c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:54.894900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626e1ee4-c604-47cd-a9b1-c2af913b7655 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:54.896352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626e1ee4-c604-47cd-a9b1-c2af913b7655 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:54.916500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 685/1374] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:39:55.006823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 690/1375] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:39:55.057173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626e1ee4-c604-47cd-a9b1-c2af913b7655 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:55.057173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-626e1ee4-c604-47cd-a9b1-c2af913b7655 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:55.149320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:55.195397 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626e1ee4-c604-47cd-a9b1-c2af913b7655 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:55.327853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35df6ce4-2d5c-49d1-a1bd-eb5b56d4757a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:55.328649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df6ce4-2d5c-49d1-a1bd-eb5b56d4757a service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:55.328943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df6ce4-2d5c-49d1-a1bd-eb5b56d4757a service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:55.329226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35df6ce4-2d5c-49d1-a1bd-eb5b56d4757a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:55.330887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-35df6ce4-2d5c-49d1-a1bd-eb5b56d4757a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:55.415921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:55.428756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35df6ce4-2d5c-49d1-a1bd-eb5b56d4757a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:55.430372 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35df6ce4-2d5c-49d1-a1bd-eb5b56d4757a service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:55.448517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 686/1376] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:39:57.490663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b353e-ab92-4f3c-8cdb-11609233095e None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:57.491236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b353e-ab92-4f3c-8cdb-11609233095e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:57.491236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4b353e-ab92-4f3c-8cdb-11609233095e None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:57.492536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 691/1377] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:39:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:39:57.506595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ea0af-06e5-427a-b05e-76accc10299d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:57.507061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ea0af-06e5-427a-b05e-76accc10299d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:57.507156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ea0af-06e5-427a-b05e-76accc10299d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:57.574702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-359ea0af-06e5-427a-b05e-76accc10299d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:39:57.576128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-359ea0af-06e5-427a-b05e-76accc10299d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:57.678043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 687/1378] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:39:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:39:57.723529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4b353e-ab92-4f3c-8cdb-11609233095e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:39:57.725239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8f4b353e-ab92-4f3c-8cdb-11609233095e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:57.777963 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:57.883851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4b353e-ab92-4f3c-8cdb-11609233095e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:39:58.005070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edb7174-7835-432b-b140-c8fde8e1327e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:39:58.005692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edb7174-7835-432b-b140-c8fde8e1327e service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:39:58.005969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edb7174-7835-432b-b140-c8fde8e1327e service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:39:58.006243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edb7174-7835-432b-b140-c8fde8e1327e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:39:58.007848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4edb7174-7835-432b-b140-c8fde8e1327e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:39:58.020394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:39:58.100877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edb7174-7835-432b-b140-c8fde8e1327e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:39:58.102409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edb7174-7835-432b-b140-c8fde8e1327e service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:39:58.123311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 692/1379] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:39:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:00.209904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420782f3-90ca-4468-a769-015eb589774d None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:00.210356 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420782f3-90ca-4468-a769-015eb589774d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:00.210467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420782f3-90ca-4468-a769-015eb589774d None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:00.211983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 688/1380] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:40:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:00.217111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499ee2a-6dfa-4753-a711-a5da05e9c34b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:00.217507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499ee2a-6dfa-4753-a711-a5da05e9c34b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:00.217756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1499ee2a-6dfa-4753-a711-a5da05e9c34b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:00.291532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-1499ee2a-6dfa-4753-a711-a5da05e9c34b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:00.292966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-1499ee2a-6dfa-4753-a711-a5da05e9c34b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:00.384437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:00.404702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 693/1381] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:40:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:40:00.448591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420782f3-90ca-4468-a769-015eb589774d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:00.450131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-420782f3-90ca-4468-a769-015eb589774d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:00.517363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:00.605923 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420782f3-90ca-4468-a769-015eb589774d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:00.742624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2cbbd4-ce90-4c44-98c4-5e74c4078c05 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:00.743410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2cbbd4-ce90-4c44-98c4-5e74c4078c05 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:00.743808 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2cbbd4-ce90-4c44-98c4-5e74c4078c05 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:00.744198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2cbbd4-ce90-4c44-98c4-5e74c4078c05 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:00.745978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-df2cbbd4-ce90-4c44-98c4-5e74c4078c05 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:00.773633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:00.890174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2cbbd4-ce90-4c44-98c4-5e74c4078c05 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:00.893170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2cbbd4-ce90-4c44-98c4-5e74c4078c05 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:00.925503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 689/1382] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 477 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:03.230617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f5386b-537c-4178-977c-ee8caec0492b None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:03.231271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f5386b-537c-4178-977c-ee8caec0492b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:03.231271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f5386b-537c-4178-977c-ee8caec0492b None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:03.232813 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 694/1383] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:40:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:03.240427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d8c118-d1af-4119-b2ba-7dc2fe97e6cb None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:03.240933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d8c118-d1af-4119-b2ba-7dc2fe97e6cb None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:03.241268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d8c118-d1af-4119-b2ba-7dc2fe97e6cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:03.330011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-74d8c118-d1af-4119-b2ba-7dc2fe97e6cb None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:03.331172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-74d8c118-d1af-4119-b2ba-7dc2fe97e6cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:03.433202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:03.441216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 690/1384] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:40:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:03.489490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f5386b-537c-4178-977c-ee8caec0492b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:03.491445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-05f5386b-537c-4178-977c-ee8caec0492b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:03.666582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f5386b-537c-4178-977c-ee8caec0492b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:03.732738 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:03.794876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b0ac54-9fac-43c3-9a89-ac117552df01 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:03.795503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b0ac54-9fac-43c3-9a89-ac117552df01 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:03.796312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b0ac54-9fac-43c3-9a89-ac117552df01 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:03.796685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b0ac54-9fac-43c3-9a89-ac117552df01 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:03.798332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-37b0ac54-9fac-43c3-9a89-ac117552df01 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:03.891344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b0ac54-9fac-43c3-9a89-ac117552df01 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:03.892976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b0ac54-9fac-43c3-9a89-ac117552df01 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:03.912061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 695/1385] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:06.013022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cd413-de4f-408f-b26f-201d3440f60e None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:06.013690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cd413-de4f-408f-b26f-201d3440f60e None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:06.013690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cd413-de4f-408f-b26f-201d3440f60e None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:06.014993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 691/1386] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:40:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:06.020719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04af431e-a716-47d7-9a51-2e5ef3ab9330 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:06.021283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04af431e-a716-47d7-9a51-2e5ef3ab9330 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:06.021283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04af431e-a716-47d7-9a51-2e5ef3ab9330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:06.105483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-04af431e-a716-47d7-9a51-2e5ef3ab9330 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:06.106725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-04af431e-a716-47d7-9a51-2e5ef3ab9330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:06.133375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:06.233092 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 696/1387] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:40:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:06.323077 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80cd413-de4f-408f-b26f-201d3440f60e None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:06.324665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e80cd413-de4f-408f-b26f-201d3440f60e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:06.514299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80cd413-de4f-408f-b26f-201d3440f60e None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:06.597858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:06.643325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301acb69-cb2d-4727-b036-a9c9a48462c4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:06.644064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301acb69-cb2d-4727-b036-a9c9a48462c4 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:06.644354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301acb69-cb2d-4727-b036-a9c9a48462c4 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:06.644670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301acb69-cb2d-4727-b036-a9c9a48462c4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:06.646335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-301acb69-cb2d-4727-b036-a9c9a48462c4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:06.745582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301acb69-cb2d-4727-b036-a9c9a48462c4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:06.747204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301acb69-cb2d-4727-b036-a9c9a48462c4 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:06.767308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 692/1388] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:08.995299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3aec15-dda7-418e-82cb-a33befc8dad1 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:08.995893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3aec15-dda7-418e-82cb-a33befc8dad1 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:08.995893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3aec15-dda7-418e-82cb-a33befc8dad1 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:08.997252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 697/1389] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:40:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:09.004283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9bf417-6040-485b-85d4-86114e2763de None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:09.004780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9bf417-6040-485b-85d4-86114e2763de None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:09.004894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9bf417-6040-485b-85d4-86114e2763de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:09.088465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-8b9bf417-6040-485b-85d4-86114e2763de None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:09.089728 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8b9bf417-6040-485b-85d4-86114e2763de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:09.126725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:09.190096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 693/1390] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:40:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:09.235731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3aec15-dda7-418e-82cb-a33befc8dad1 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:09.237332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8c3aec15-dda7-418e-82cb-a33befc8dad1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:09.411589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3aec15-dda7-418e-82cb-a33befc8dad1 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:09.421620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:09.526805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac0d2ff-79e3-4a02-a4c0-5ffc94ba24c2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:09.527520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac0d2ff-79e3-4a02-a4c0-5ffc94ba24c2 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:09.527803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac0d2ff-79e3-4a02-a4c0-5ffc94ba24c2 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:09.528077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac0d2ff-79e3-4a02-a4c0-5ffc94ba24c2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:09.529744 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fac0d2ff-79e3-4a02-a4c0-5ffc94ba24c2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:09.622839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac0d2ff-79e3-4a02-a4c0-5ffc94ba24c2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:09.625019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac0d2ff-79e3-4a02-a4c0-5ffc94ba24c2 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:09.643283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 698/1391] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:10.157858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cf7c97-d21a-4cd2-8492-4aa4513a13ed None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:10.158449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cf7c97-d21a-4cd2-8492-4aa4513a13ed None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:10.158449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99cf7c97-d21a-4cd2-8492-4aa4513a13ed None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:10.159888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 694/1392] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:40:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:10.165233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ab38ef-ca91-4654-be5d-962879de6530 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:10.165680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ab38ef-ca91-4654-be5d-962879de6530 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:10.165785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ab38ef-ca91-4654-be5d-962879de6530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:10.181597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:10.235486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f5ab38ef-ca91-4654-be5d-962879de6530 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:10.236743 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f5ab38ef-ca91-4654-be5d-962879de6530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:10.338278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 699/1393] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:40:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:10.394265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab5a4e9-3668-4f07-969f-de66dc662632 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:10.394560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab5a4e9-3668-4f07-969f-de66dc662632 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:10.394901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab5a4e9-3668-4f07-969f-de66dc662632 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:10.396296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 695/1394] 10.222.0.40 () {54 vars in 966 bytes} [Fri Jul 29 04:40:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:10.401414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f831dafc-30c6-4624-a4b0-b4483a902655 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:10.401784 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f831dafc-30c6-4624-a4b0-b4483a902655 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:10.402153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f831dafc-30c6-4624-a4b0-b4483a902655 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:10.478808 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f831dafc-30c6-4624-a4b0-b4483a902655 None None] MFA Rules not processed for user `3749dc021aac478f88072ced48f4cee4`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:10.479936 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f831dafc-30c6-4624-a4b0-b4483a902655 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:10.516969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:10.559128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 700/1395] 10.222.0.40 () {58 vars in 1063 bytes} [Fri Jul 29 04:40:10 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:10.567869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab5a4e9-3668-4f07-969f-de66dc662632 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:10.569441 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-4ab5a4e9-3668-4f07-969f-de66dc662632 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:10.679707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab5a4e9-3668-4f07-969f-de66dc662632 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:10.744273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:10.757324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7110a56d-4cd2-461f-a2ad-da6fd2cfcf5f service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3749dc021aac478f88072ced48f4cee4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:10.757952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7110a56d-4cd2-461f-a2ad-da6fd2cfcf5f service glance] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:10.758240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7110a56d-4cd2-461f-a2ad-da6fd2cfcf5f service glance] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:10.758523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7110a56d-4cd2-461f-a2ad-da6fd2cfcf5f service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:10.760195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7110a56d-4cd2-461f-a2ad-da6fd2cfcf5f service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:10.878243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7110a56d-4cd2-461f-a2ad-da6fd2cfcf5f service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:10.879690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7110a56d-4cd2-461f-a2ad-da6fd2cfcf5f service glance] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:10.897588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 696/1396] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:40:11.702834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a90db-da5e-4bcc-b4bc-f30e64d7c5ca None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:11.703379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a90db-da5e-4bcc-b4bc-f30e64d7c5ca None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:11.703472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a90db-da5e-4bcc-b4bc-f30e64d7c5ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:11.767225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d94e4e-a4b6-4001-a118-67819032941c None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:11.767854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d94e4e-a4b6-4001-a118-67819032941c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:11.767854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d94e4e-a4b6-4001-a118-67819032941c None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:11.769361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 697/1397] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:40:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:11.775048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a76fcc3-a5d7-43a2-85ee-904cb0956bc6 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:11.775048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a76fcc3-a5d7-43a2-85ee-904cb0956bc6 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:11.775667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a76fcc3-a5d7-43a2-85ee-904cb0956bc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:11.795002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-aa8a90db-da5e-4bcc-b4bc-f30e64d7c5ca None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:11.796949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-aa8a90db-da5e-4bcc-b4bc-f30e64d7c5ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:11.850329 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9a76fcc3-a5d7-43a2-85ee-904cb0956bc6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:11.851600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9a76fcc3-a5d7-43a2-85ee-904cb0956bc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:11.895053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:11.902849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 701/1398] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:40:11 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:11.951569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5a7aa-3a46-4370-a12b-ec02f93577f0 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:11.952696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5a7aa-3a46-4370-a12b-ec02f93577f0 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:11.953168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5a7aa-3a46-4370-a12b-ec02f93577f0 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:11.955237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 702/1399] 10.222.0.40 () {54 vars in 943 bytes} [Fri Jul 29 04:40:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:40:11.962067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a8e29a-5e67-4e97-b530-676d78242f6b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:11.962431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a8e29a-5e67-4e97-b530-676d78242f6b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:11.962719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a8e29a-5e67-4e97-b530-676d78242f6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:11.967271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 698/1400] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:40:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:40:12.012281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a76fcc3-a5d7-43a2-85ee-904cb0956bc6 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:12.013539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9a76fcc3-a5d7-43a2-85ee-904cb0956bc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:12.026369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:12.047455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-b6a8e29a-5e67-4e97-b530-676d78242f6b None None] MFA Rules not processed for user `bbfe92babca048d882cb59cb29b3ed97`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:12.048715 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b6a8e29a-5e67-4e97-b530-676d78242f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:12.135888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a76fcc3-a5d7-43a2-85ee-904cb0956bc6 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:12.176699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 703/1401] 10.222.0.40 () {58 vars in 1040 bytes} [Fri Jul 29 04:40:11 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:40:12.185824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a8e29a-5e67-4e97-b530-676d78242f6b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:12.196968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a8e29a-5e67-4e97-b530-676d78242f6b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:12.215090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:12.282841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff316bb-e30e-4a68-950a-8f9d5efe3d5a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:12.283478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff316bb-e30e-4a68-950a-8f9d5efe3d5a service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:12.283764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff316bb-e30e-4a68-950a-8f9d5efe3d5a service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:12.284042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff316bb-e30e-4a68-950a-8f9d5efe3d5a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:12.285742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7ff316bb-e30e-4a68-950a-8f9d5efe3d5a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:12.317526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:12.323639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bb4672-291d-4edd-8100-88e74ddf42a6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:12.324271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bb4672-291d-4edd-8100-88e74ddf42a6 service swift] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:12.324558 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bb4672-291d-4edd-8100-88e74ddf42a6 service swift] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:12.324872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bb4672-291d-4edd-8100-88e74ddf42a6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:12.326585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-18bb4672-291d-4edd-8100-88e74ddf42a6 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:12.386210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff316bb-e30e-4a68-950a-8f9d5efe3d5a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:12.387846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff316bb-e30e-4a68-950a-8f9d5efe3d5a service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:12.407895 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 699/1402] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:40:12.419740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bb4672-291d-4edd-8100-88e74ddf42a6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:12.422068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bb4672-291d-4edd-8100-88e74ddf42a6 service swift] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:12.432391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 704/1403] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:40:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 29 04:40:14.586911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd899fc7-c746-4d61-aca0-6a1fe323eee9 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:14.587794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd899fc7-c746-4d61-aca0-6a1fe323eee9 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:14.588396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd899fc7-c746-4d61-aca0-6a1fe323eee9 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:14.599511 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2421ea50-6cd5-4aef-95f5-5ef78fa03198 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:14.600209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2421ea50-6cd5-4aef-95f5-5ef78fa03198 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:14.600826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2421ea50-6cd5-4aef-95f5-5ef78fa03198 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:14.606655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 700/1404] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:40:14 2022] GET /identity => generated 270 bytes in 22 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:14.694524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-2421ea50-6cd5-4aef-95f5-5ef78fa03198 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:14.696026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2421ea50-6cd5-4aef-95f5-5ef78fa03198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:14.799195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:14.813870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 705/1405] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:40:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:14.859470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd899fc7-c746-4d61-aca0-6a1fe323eee9 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:14.861470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-cd899fc7-c746-4d61-aca0-6a1fe323eee9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:14.958785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:15.081838 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd899fc7-c746-4d61-aca0-6a1fe323eee9 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:15.234101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f217d2-cfe5-46cd-9a1a-272537e605b4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:15.234740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f217d2-cfe5-46cd-9a1a-272537e605b4 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:15.235024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f217d2-cfe5-46cd-9a1a-272537e605b4 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:15.235304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f217d2-cfe5-46cd-9a1a-272537e605b4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:15.239199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-99f217d2-cfe5-46cd-9a1a-272537e605b4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:15.281476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:15.355549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f217d2-cfe5-46cd-9a1a-272537e605b4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:15.357381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f217d2-cfe5-46cd-9a1a-272537e605b4 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:15.381711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 701/1406] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:17.576443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db6b2b-8db8-49c7-bb08-d3c9ac58f25f None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:17.577221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db6b2b-8db8-49c7-bb08-d3c9ac58f25f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:17.577221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db6b2b-8db8-49c7-bb08-d3c9ac58f25f None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:17.578455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 706/1407] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:40:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:17.583611 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4d9b1-abae-4ecb-88d5-49d797b6bec4 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:17.584042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4d9b1-abae-4ecb-88d5-49d797b6bec4 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:17.584368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d4d9b1-abae-4ecb-88d5-49d797b6bec4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:17.669382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d7d4d9b1-abae-4ecb-88d5-49d797b6bec4 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:17.670654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d7d4d9b1-abae-4ecb-88d5-49d797b6bec4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:17.784536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 702/1408] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:40:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:40:17.829131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18db6b2b-8db8-49c7-bb08-d3c9ac58f25f None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:17.830962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-18db6b2b-8db8-49c7-bb08-d3c9ac58f25f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:17.995705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18db6b2b-8db8-49c7-bb08-d3c9ac58f25f None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:18.069248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:18.128933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7689e96e-79be-4afe-b144-8aa4e180bf30 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:18.130073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689e96e-79be-4afe-b144-8aa4e180bf30 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:18.130536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689e96e-79be-4afe-b144-8aa4e180bf30 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:18.130984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689e96e-79be-4afe-b144-8aa4e180bf30 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:18.132686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7689e96e-79be-4afe-b144-8aa4e180bf30 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:18.229787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7689e96e-79be-4afe-b144-8aa4e180bf30 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:18.231439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7689e96e-79be-4afe-b144-8aa4e180bf30 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:18.253203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 707/1409] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:27.781935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=98 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:28.250853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=67 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:35.311491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e816922d-0651-4ad5-bc12-c06cfc7525dc None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:35.312051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e816922d-0651-4ad5-bc12-c06cfc7525dc None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:35.312286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e816922d-0651-4ad5-bc12-c06cfc7525dc None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:35.313713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 703/1410] 10.222.0.40 () {54 vars in 918 bytes} [Fri Jul 29 04:40:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:35.318818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7760e098-41db-4894-a44b-28fd7447f2e4 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:35.319270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7760e098-41db-4894-a44b-28fd7447f2e4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:35.319371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7760e098-41db-4894-a44b-28fd7447f2e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:35.394798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-7760e098-41db-4894-a44b-28fd7447f2e4 None None] MFA Rules not processed for user `e3da5e428049419a9b35494655f52484`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:35.396005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7760e098-41db-4894-a44b-28fd7447f2e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:35.502628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 708/1411] 10.222.0.40 () {58 vars in 1015 bytes} [Fri Jul 29 04:40:35 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:40:35.517804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2194f264-5949-4b53-9515-1d40827498a4 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:35.518100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2194f264-5949-4b53-9515-1d40827498a4 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:35.518388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2194f264-5949-4b53-9515-1d40827498a4 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:35.519745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 704/1412] 10.222.0.40 () {54 vars in 918 bytes} [Fri Jul 29 04:40:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:35.524243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bceb2a-62e9-49dd-aa77-76a0e7b999a9 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:35.524539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bceb2a-62e9-49dd-aa77-76a0e7b999a9 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:35.524858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bceb2a-62e9-49dd-aa77-76a0e7b999a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:35.597654 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:35.600325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-27bceb2a-62e9-49dd-aa77-76a0e7b999a9 None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:35.602059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-27bceb2a-62e9-49dd-aa77-76a0e7b999a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:35.716719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 709/1413] 10.222.0.40 () {58 vars in 1015 bytes} [Fri Jul 29 04:40:35 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:38.215506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3357a088-80e3-4dc3-8f15-a168b45d6a79 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:38.216041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3357a088-80e3-4dc3-8f15-a168b45d6a79 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:38.216041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3357a088-80e3-4dc3-8f15-a168b45d6a79 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:38.217391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 705/1414] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:40:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:38.223257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55eec975-730d-41d8-b435-a237e3ec2e42 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:38.223660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55eec975-730d-41d8-b435-a237e3ec2e42 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:38.223812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55eec975-730d-41d8-b435-a237e3ec2e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:38.290075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-55eec975-730d-41d8-b435-a237e3ec2e42 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:38.291315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-55eec975-730d-41d8-b435-a237e3ec2e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:38.345539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:38.387120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 710/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:40:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:38.430930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3357a088-80e3-4dc3-8f15-a168b45d6a79 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:38.432456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3357a088-80e3-4dc3-8f15-a168b45d6a79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:38.577949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3357a088-80e3-4dc3-8f15-a168b45d6a79 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:38.650487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:38.692439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c62016-e973-4589-a4b0-b7beb17d247c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:38.693096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c62016-e973-4589-a4b0-b7beb17d247c service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:38.693384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c62016-e973-4589-a4b0-b7beb17d247c service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:38.693665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c62016-e973-4589-a4b0-b7beb17d247c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:38.695308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d8c62016-e973-4589-a4b0-b7beb17d247c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:38.793174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c62016-e973-4589-a4b0-b7beb17d247c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:38.795051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c62016-e973-4589-a4b0-b7beb17d247c service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:38.813821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 706/1416] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:39.715556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9625dc0b-aa42-4629-927c-73ccbecfcce5 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:39.716109 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9625dc0b-aa42-4629-927c-73ccbecfcce5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:39.716109 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9625dc0b-aa42-4629-927c-73ccbecfcce5 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:39.717509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 711/1417] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:40:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:39.722504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3209a7bb-ad76-495d-baf5-8bba67b37088 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:39.722948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3209a7bb-ad76-495d-baf5-8bba67b37088 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:39.723060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3209a7bb-ad76-495d-baf5-8bba67b37088 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:39.799345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-3209a7bb-ad76-495d-baf5-8bba67b37088 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:39.800625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3209a7bb-ad76-495d-baf5-8bba67b37088 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:39.827545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:39.896620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 707/1418] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:40:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:40.047085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9625dc0b-aa42-4629-927c-73ccbecfcce5 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:40.048648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9625dc0b-aa42-4629-927c-73ccbecfcce5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:40.193568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9625dc0b-aa42-4629-927c-73ccbecfcce5 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:40.232408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:40.305532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48f6b67-0d45-4722-b23c-028af80ced62 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:40.306133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48f6b67-0d45-4722-b23c-028af80ced62 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:40.306417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48f6b67-0d45-4722-b23c-028af80ced62 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:40.306692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48f6b67-0d45-4722-b23c-028af80ced62 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:40.308308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d48f6b67-0d45-4722-b23c-028af80ced62 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:40.397698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48f6b67-0d45-4722-b23c-028af80ced62 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:40.399333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48f6b67-0d45-4722-b23c-028af80ced62 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:40.417960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 712/1419] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:41.309294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9c1a83-233a-4cab-8b6f-859bf36cf81b None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:41.309853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9c1a83-233a-4cab-8b6f-859bf36cf81b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:41.309853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9c1a83-233a-4cab-8b6f-859bf36cf81b None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:41.311219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 708/1420] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:40:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:41.316450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9024193-5b15-4ee6-a4da-ff72d45b5697 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:41.316940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9024193-5b15-4ee6-a4da-ff72d45b5697 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:41.317064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9024193-5b15-4ee6-a4da-ff72d45b5697 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:41.371736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:41.395646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-d9024193-5b15-4ee6-a4da-ff72d45b5697 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:41.397706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d9024193-5b15-4ee6-a4da-ff72d45b5697 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:41.496406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 713/1421] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:40:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:41.658278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9c1a83-233a-4cab-8b6f-859bf36cf81b None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:41.659855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5b9c1a83-233a-4cab-8b6f-859bf36cf81b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:41.807239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9c1a83-233a-4cab-8b6f-859bf36cf81b None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:41.819575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:41.920413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528e27b7-be85-47e5-8c92-00f9f424e580 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:41.921738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528e27b7-be85-47e5-8c92-00f9f424e580 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:41.922029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528e27b7-be85-47e5-8c92-00f9f424e580 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:41.922310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528e27b7-be85-47e5-8c92-00f9f424e580 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:41.923950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-528e27b7-be85-47e5-8c92-00f9f424e580 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:42.011444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528e27b7-be85-47e5-8c92-00f9f424e580 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:42.013079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528e27b7-be85-47e5-8c92-00f9f424e580 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:42.031560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 709/1422] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:42.881174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8613afeb-5805-49c4-8100-a384178ece82 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:42.881743 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8613afeb-5805-49c4-8100-a384178ece82 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:42.881743 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8613afeb-5805-49c4-8100-a384178ece82 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:42.883107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 714/1423] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:40:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:42.888170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49806e-75b4-4850-b581-1e9b51a35a5c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:42.888588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49806e-75b4-4850-b581-1e9b51a35a5c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:42.888787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed49806e-75b4-4850-b581-1e9b51a35a5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:42.908464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:42.965766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-ed49806e-75b4-4850-b581-1e9b51a35a5c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:42.967005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ed49806e-75b4-4850-b581-1e9b51a35a5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:43.067706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 710/1424] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:40:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:43.220701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8613afeb-5805-49c4-8100-a384178ece82 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:43.222208 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8613afeb-5805-49c4-8100-a384178ece82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:43.335145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:43.372511 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8613afeb-5805-49c4-8100-a384178ece82 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:43.487193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6305ea48-a683-485e-bd7a-fb091da909bf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:43.487824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305ea48-a683-485e-bd7a-fb091da909bf service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:43.488108 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305ea48-a683-485e-bd7a-fb091da909bf service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:43.488385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6305ea48-a683-485e-bd7a-fb091da909bf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:43.490071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6305ea48-a683-485e-bd7a-fb091da909bf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:43.566214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:43.580066 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6305ea48-a683-485e-bd7a-fb091da909bf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:43.581732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6305ea48-a683-485e-bd7a-fb091da909bf service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:43.601751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 715/1425] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:44.464832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f84420-4287-43e8-89b8-0f115ca84649 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:44.465408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f84420-4287-43e8-89b8-0f115ca84649 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:44.465408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f84420-4287-43e8-89b8-0f115ca84649 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:44.466773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 711/1426] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:40:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:44.472472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5e096-b11a-410d-b9f3-e167d42b0d68 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:44.472951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5e096-b11a-410d-b9f3-e167d42b0d68 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:44.473048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab5e096-b11a-410d-b9f3-e167d42b0d68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:44.548444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-7ab5e096-b11a-410d-b9f3-e167d42b0d68 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:44.549685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7ab5e096-b11a-410d-b9f3-e167d42b0d68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:44.645472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 716/1427] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:40:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:44.795261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f84420-4287-43e8-89b8-0f115ca84649 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:44.796869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-05f84420-4287-43e8-89b8-0f115ca84649 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:44.880772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:44.943352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f84420-4287-43e8-89b8-0f115ca84649 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:45.054460 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a465a6f-34bb-4201-a109-627bd264ba95 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:45.054998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a465a6f-34bb-4201-a109-627bd264ba95 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:45.055279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a465a6f-34bb-4201-a109-627bd264ba95 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:45.055548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a465a6f-34bb-4201-a109-627bd264ba95 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:45.057183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0a465a6f-34bb-4201-a109-627bd264ba95 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:45.105340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:45.152738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a465a6f-34bb-4201-a109-627bd264ba95 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:45.154357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a465a6f-34bb-4201-a109-627bd264ba95 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:45.173492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 712/1428] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:45.973805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb83e3c-6828-4b5b-acb3-d2b43173c24e None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:45.974347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb83e3c-6828-4b5b-acb3-d2b43173c24e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:45.974347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb83e3c-6828-4b5b-acb3-d2b43173c24e None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:45.975657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 717/1429] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:40:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:45.980433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4580801c-243c-4613-b36e-b1a663e0cee1 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:45.980897 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4580801c-243c-4613-b36e-b1a663e0cee1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:45.981011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4580801c-243c-4613-b36e-b1a663e0cee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:46.055407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-4580801c-243c-4613-b36e-b1a663e0cee1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:46.056675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-4580801c-243c-4613-b36e-b1a663e0cee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:46.153277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 713/1430] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:40:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:46.305236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb83e3c-6828-4b5b-acb3-d2b43173c24e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:46.306777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8fb83e3c-6828-4b5b-acb3-d2b43173c24e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:46.356552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:46.455559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb83e3c-6828-4b5b-acb3-d2b43173c24e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:46.567603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea166fc0-2f97-44be-9bce-ec65982f080f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:46.568143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea166fc0-2f97-44be-9bce-ec65982f080f service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:46.568428 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea166fc0-2f97-44be-9bce-ec65982f080f service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:46.568758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea166fc0-2f97-44be-9bce-ec65982f080f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:46.570394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ea166fc0-2f97-44be-9bce-ec65982f080f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:46.577993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:46.659653 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea166fc0-2f97-44be-9bce-ec65982f080f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:46.661351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea166fc0-2f97-44be-9bce-ec65982f080f service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:46.679525 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 718/1431] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:47.494723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a67ff1-c9ac-4284-bb8d-1f98ec4e5957 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:47.495270 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a67ff1-c9ac-4284-bb8d-1f98ec4e5957 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:47.495270 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a67ff1-c9ac-4284-bb8d-1f98ec4e5957 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:47.496692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 714/1432] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:40:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:47.502505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64de344e-dca5-4da7-b7e4-e011a37400fc None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:47.502976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64de344e-dca5-4da7-b7e4-e011a37400fc None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:47.503071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64de344e-dca5-4da7-b7e4-e011a37400fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:47.580026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-64de344e-dca5-4da7-b7e4-e011a37400fc None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:47.581275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-64de344e-dca5-4da7-b7e4-e011a37400fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:47.648059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:47.679747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 719/1433] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:40:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:47.832568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a67ff1-c9ac-4284-bb8d-1f98ec4e5957 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:47.834040 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c7a67ff1-c9ac-4284-bb8d-1f98ec4e5957 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:47.851984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:47.977805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a67ff1-c9ac-4284-bb8d-1f98ec4e5957 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:48.063046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:48.089300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d49cad2-e27d-4a0c-892d-449d20510695 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:48.089858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d49cad2-e27d-4a0c-892d-449d20510695 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:48.090139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d49cad2-e27d-4a0c-892d-449d20510695 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:48.090411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d49cad2-e27d-4a0c-892d-449d20510695 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:48.092020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7d49cad2-e27d-4a0c-892d-449d20510695 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:48.181096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d49cad2-e27d-4a0c-892d-449d20510695 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:48.182726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d49cad2-e27d-4a0c-892d-449d20510695 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:48.200747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 715/1434] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:49.020583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a9219b-37e6-4ab1-91b5-8ed35a69a82d None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:49.021165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a9219b-37e6-4ab1-91b5-8ed35a69a82d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:49.021268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a9219b-37e6-4ab1-91b5-8ed35a69a82d None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:49.023061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 720/1435] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:40:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:49.028648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81df23d-e14c-43b2-bb52-c70d3bdc720d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:49.029094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81df23d-e14c-43b2-bb52-c70d3bdc720d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:49.029302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81df23d-e14c-43b2-bb52-c70d3bdc720d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:49.239693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c81df23d-e14c-43b2-bb52-c70d3bdc720d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:49.240940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c81df23d-e14c-43b2-bb52-c70d3bdc720d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:49.282236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:49.337707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 716/1436] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:40:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:49.489119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a9219b-37e6-4ab1-91b5-8ed35a69a82d None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:49.490664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-89a9219b-37e6-4ab1-91b5-8ed35a69a82d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:49.645674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a9219b-37e6-4ab1-91b5-8ed35a69a82d None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:49.701400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:49.760776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a964a638-0282-4bba-bf2b-c5da775d2112 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:49.761400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a964a638-0282-4bba-bf2b-c5da775d2112 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:49.761686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a964a638-0282-4bba-bf2b-c5da775d2112 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:49.761966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a964a638-0282-4bba-bf2b-c5da775d2112 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:49.763602 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a964a638-0282-4bba-bf2b-c5da775d2112 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:49.854779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a964a638-0282-4bba-bf2b-c5da775d2112 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:49.856370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a964a638-0282-4bba-bf2b-c5da775d2112 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:49.874741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 721/1437] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:50.681158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c174e6e-220e-4bbc-a7fd-99e39dce3681 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:50.681618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c174e6e-220e-4bbc-a7fd-99e39dce3681 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:50.681715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c174e6e-220e-4bbc-a7fd-99e39dce3681 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:50.683118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 717/1438] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:40:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:50.688221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343c0b9-1d09-4c13-85d7-48d657493aaf None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:50.688676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343c0b9-1d09-4c13-85d7-48d657493aaf None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:50.688828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343c0b9-1d09-4c13-85d7-48d657493aaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:50.763465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c343c0b9-1d09-4c13-85d7-48d657493aaf None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:50.764819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c343c0b9-1d09-4c13-85d7-48d657493aaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:50.766857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:50.861548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 722/1439] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:40:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:51.012108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c174e6e-220e-4bbc-a7fd-99e39dce3681 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:51.013710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2c174e6e-220e-4bbc-a7fd-99e39dce3681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:51.159492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c174e6e-220e-4bbc-a7fd-99e39dce3681 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:51.187956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:51.274222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c02c7d-14d1-4ff0-9c2b-0cbda9086713 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:51.274775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c02c7d-14d1-4ff0-9c2b-0cbda9086713 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:51.275067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c02c7d-14d1-4ff0-9c2b-0cbda9086713 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:51.275345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c02c7d-14d1-4ff0-9c2b-0cbda9086713 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:51.277008 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b4c02c7d-14d1-4ff0-9c2b-0cbda9086713 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:51.366656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c02c7d-14d1-4ff0-9c2b-0cbda9086713 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:51.368223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c02c7d-14d1-4ff0-9c2b-0cbda9086713 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:51.386362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 718/1440] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:53.276052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f639a1-584e-4fdb-88c6-f1a9a3c15a89 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:53.276742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f639a1-584e-4fdb-88c6-f1a9a3c15a89 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:53.276742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f639a1-584e-4fdb-88c6-f1a9a3c15a89 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:53.278577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 723/1441] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:40:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:53.284089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c3f6fe-3c38-49b3-9cf6-ef3d00e2c265 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:53.284553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c3f6fe-3c38-49b3-9cf6-ef3d00e2c265 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:53.284717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c3f6fe-3c38-49b3-9cf6-ef3d00e2c265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:53.326235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:53.352721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-65c3f6fe-3c38-49b3-9cf6-ef3d00e2c265 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:53.353892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-65c3f6fe-3c38-49b3-9cf6-ef3d00e2c265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:53.451754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 719/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:40:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:53.490277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f639a1-584e-4fdb-88c6-f1a9a3c15a89 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:53.491832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-54f639a1-584e-4fdb-88c6-f1a9a3c15a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:53.597659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f639a1-584e-4fdb-88c6-f1a9a3c15a89 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:53.630077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:53.672816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a61dcd-fce5-430e-91d0-9980794b8a57 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:53.673442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a61dcd-fce5-430e-91d0-9980794b8a57 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:53.673727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a61dcd-fce5-430e-91d0-9980794b8a57 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:53.674005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a61dcd-fce5-430e-91d0-9980794b8a57 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:53.675654 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-85a61dcd-fce5-430e-91d0-9980794b8a57 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:53.783303 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a61dcd-fce5-430e-91d0-9980794b8a57 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:53.784786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a61dcd-fce5-430e-91d0-9980794b8a57 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:53.803436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 724/1443] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:56.007342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664232e4-8b76-4d67-8f62-216905ece7b5 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:56.007974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664232e4-8b76-4d67-8f62-216905ece7b5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:56.007974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664232e4-8b76-4d67-8f62-216905ece7b5 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:56.009281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 720/1444] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:40:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:56.014409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d6139d-0f3d-4463-809d-c526b5733561 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:56.014848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d6139d-0f3d-4463-809d-c526b5733561 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:56.014985 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d6139d-0f3d-4463-809d-c526b5733561 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:56.080274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-d1d6139d-0f3d-4463-809d-c526b5733561 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:56.081498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d1d6139d-0f3d-4463-809d-c526b5733561 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:56.094670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:56.179944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 725/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:40:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:56.219391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664232e4-8b76-4d67-8f62-216905ece7b5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:56.220985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-664232e4-8b76-4d67-8f62-216905ece7b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:56.328489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664232e4-8b76-4d67-8f62-216905ece7b5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:56.340293 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:56.401284 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab61250-6112-4f7b-91fd-2cba96a09ead service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:56.401902 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab61250-6112-4f7b-91fd-2cba96a09ead service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:56.402187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab61250-6112-4f7b-91fd-2cba96a09ead service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:56.402464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab61250-6112-4f7b-91fd-2cba96a09ead service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:56.404088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3ab61250-6112-4f7b-91fd-2cba96a09ead service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:56.513168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab61250-6112-4f7b-91fd-2cba96a09ead service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:56.516754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab61250-6112-4f7b-91fd-2cba96a09ead service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:56.535982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 721/1446] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:40:58.826684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de821210-fe8c-4d03-acf2-cc9c1fe06a7e None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:58.827866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de821210-fe8c-4d03-acf2-cc9c1fe06a7e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:58.827866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de821210-fe8c-4d03-acf2-cc9c1fe06a7e None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:58.829366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 726/1447] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:40:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:40:58.834700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ecd208-de0a-4d3a-a09c-d8d3d54a342c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:58.835115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ecd208-de0a-4d3a-a09c-d8d3d54a342c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:58.835262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ecd208-de0a-4d3a-a09c-d8d3d54a342c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:58.883229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:58.901600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-46ecd208-de0a-4d3a-a09c-d8d3d54a342c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:40:58.902799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-46ecd208-de0a-4d3a-a09c-d8d3d54a342c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:58.998145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 722/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:40:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:40:59.037553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de821210-fe8c-4d03-acf2-cc9c1fe06a7e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:40:59.039096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-de821210-fe8c-4d03-acf2-cc9c1fe06a7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:59.142968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de821210-fe8c-4d03-acf2-cc9c1fe06a7e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:40:59.182379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:40:59.218623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63c4c62-1818-471d-8e3f-fa77cf772fce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:40:59.219250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63c4c62-1818-471d-8e3f-fa77cf772fce service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:40:59.219538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63c4c62-1818-471d-8e3f-fa77cf772fce service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:40:59.219817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63c4c62-1818-471d-8e3f-fa77cf772fce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:40:59.221501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b63c4c62-1818-471d-8e3f-fa77cf772fce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:40:59.332577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63c4c62-1818-471d-8e3f-fa77cf772fce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:40:59.334037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b63c4c62-1818-471d-8e3f-fa77cf772fce service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:40:59.351741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 727/1449] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:40:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:01.466081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297d297-1b50-4836-ae7c-4b034cf15c03 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:01.466600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297d297-1b50-4836-ae7c-4b034cf15c03 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:01.466600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5297d297-1b50-4836-ae7c-4b034cf15c03 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:01.468669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 723/1450] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:41:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:01.474021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8094c31-0c80-4dd9-9b0a-224b5f7c11a0 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:01.474497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8094c31-0c80-4dd9-9b0a-224b5f7c11a0 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:01.474606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8094c31-0c80-4dd9-9b0a-224b5f7c11a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:01.541611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e8094c31-0c80-4dd9-9b0a-224b5f7c11a0 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:01.542854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e8094c31-0c80-4dd9-9b0a-224b5f7c11a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:01.564990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:01.641290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 728/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:41:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:41:01.680889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297d297-1b50-4836-ae7c-4b034cf15c03 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:01.682448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5297d297-1b50-4836-ae7c-4b034cf15c03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:01.787003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5297d297-1b50-4836-ae7c-4b034cf15c03 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:01.806982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:01.860940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05baf37c-87f2-4760-8040-5eebf07b9c87 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:01.861586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05baf37c-87f2-4760-8040-5eebf07b9c87 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:01.861889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05baf37c-87f2-4760-8040-5eebf07b9c87 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:01.862187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05baf37c-87f2-4760-8040-5eebf07b9c87 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:01.863821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-05baf37c-87f2-4760-8040-5eebf07b9c87 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:01.971999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05baf37c-87f2-4760-8040-5eebf07b9c87 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:01.973467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05baf37c-87f2-4760-8040-5eebf07b9c87 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:01.991158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 724/1452] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:04.049259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29976e10-b86d-4543-8540-f0f20e12fca3 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:04.049824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29976e10-b86d-4543-8540-f0f20e12fca3 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:04.049930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29976e10-b86d-4543-8540-f0f20e12fca3 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:04.051222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 729/1453] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:41:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:04.056354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6dd016-aa14-40cb-ba03-61bded4fb9b5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:04.056821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6dd016-aa14-40cb-ba03-61bded4fb9b5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:04.056952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6dd016-aa14-40cb-ba03-61bded4fb9b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:04.120099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:04.122772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-fa6dd016-aa14-40cb-ba03-61bded4fb9b5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:04.125110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fa6dd016-aa14-40cb-ba03-61bded4fb9b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:04.221683 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 725/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:41:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:04.259711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29976e10-b86d-4543-8540-f0f20e12fca3 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:04.261350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-29976e10-b86d-4543-8540-f0f20e12fca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:04.363765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29976e10-b86d-4543-8540-f0f20e12fca3 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:04.412316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:04.438279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5365033-384b-4bf8-ab15-c11edb5d9c37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:04.438893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5365033-384b-4bf8-ab15-c11edb5d9c37 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:04.439181 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5365033-384b-4bf8-ab15-c11edb5d9c37 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:04.439459 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5365033-384b-4bf8-ab15-c11edb5d9c37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:04.441117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d5365033-384b-4bf8-ab15-c11edb5d9c37 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:04.550617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5365033-384b-4bf8-ab15-c11edb5d9c37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:04.552060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5365033-384b-4bf8-ab15-c11edb5d9c37 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:04.570524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 730/1455] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:06.621678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fe903f-11b0-4681-ba23-447ecfb7654f None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:06.622256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fe903f-11b0-4681-ba23-447ecfb7654f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:06.622256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fe903f-11b0-4681-ba23-447ecfb7654f None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:06.624275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 726/1456] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:41:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:06.629484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ceb77-930a-4d4f-8895-9cc8bc682c6b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:06.629912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ceb77-930a-4d4f-8895-9cc8bc682c6b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:06.630070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7ceb77-930a-4d4f-8895-9cc8bc682c6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:06.818315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-5f7ceb77-930a-4d4f-8895-9cc8bc682c6b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:06.819521 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5f7ceb77-930a-4d4f-8895-9cc8bc682c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:06.847351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:06.916374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 731/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:41:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:06.955318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fe903f-11b0-4681-ba23-447ecfb7654f None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:06.957004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-34fe903f-11b0-4681-ba23-447ecfb7654f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:07.058174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fe903f-11b0-4681-ba23-447ecfb7654f None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:07.086557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:07.133838 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef6fb8b-79b0-4a05-a2b5-632572095f8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:07.134457 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef6fb8b-79b0-4a05-a2b5-632572095f8d service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:07.134743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef6fb8b-79b0-4a05-a2b5-632572095f8d service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:07.135024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef6fb8b-79b0-4a05-a2b5-632572095f8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:07.136670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8ef6fb8b-79b0-4a05-a2b5-632572095f8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:07.247236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef6fb8b-79b0-4a05-a2b5-632572095f8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:07.249789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef6fb8b-79b0-4a05-a2b5-632572095f8d service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:07.275540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 727/1458] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:09.345443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064aca50-775d-4342-b17b-23dd1ef11902 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:09.345969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064aca50-775d-4342-b17b-23dd1ef11902 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:09.345969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064aca50-775d-4342-b17b-23dd1ef11902 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:09.347299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 732/1459] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:41:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:09.352812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d697250b-0d34-4f91-8eb4-23bb611bb3b5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:09.353283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d697250b-0d34-4f91-8eb4-23bb611bb3b5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:09.353398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d697250b-0d34-4f91-8eb4-23bb611bb3b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:09.424473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d697250b-0d34-4f91-8eb4-23bb611bb3b5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:09.425828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d697250b-0d34-4f91-8eb4-23bb611bb3b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:09.435476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:09.548245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 728/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:41:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:09.591276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064aca50-775d-4342-b17b-23dd1ef11902 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:09.593020 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-064aca50-775d-4342-b17b-23dd1ef11902 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:09.709974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064aca50-775d-4342-b17b-23dd1ef11902 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:09.779435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:09.799150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af91d4ca-1c97-4a97-9820-7e440e7fa2f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:09.799777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91d4ca-1c97-4a97-9820-7e440e7fa2f8 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:09.800060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91d4ca-1c97-4a97-9820-7e440e7fa2f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:09.800336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91d4ca-1c97-4a97-9820-7e440e7fa2f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:09.802027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-af91d4ca-1c97-4a97-9820-7e440e7fa2f8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:09.922084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af91d4ca-1c97-4a97-9820-7e440e7fa2f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:09.923436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af91d4ca-1c97-4a97-9820-7e440e7fa2f8 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:09.944469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 733/1461] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:12.019752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd91a1ff-f8e2-42d2-9223-0015c3d4ee31 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:12.020275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd91a1ff-f8e2-42d2-9223-0015c3d4ee31 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:12.020400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd91a1ff-f8e2-42d2-9223-0015c3d4ee31 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:12.021696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 729/1462] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:41:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:12.026989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec21eb8-4654-4eb0-8a12-27c3361d800e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:12.027422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec21eb8-4654-4eb0-8a12-27c3361d800e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:12.027604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec21eb8-4654-4eb0-8a12-27c3361d800e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:12.096772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-fec21eb8-4654-4eb0-8a12-27c3361d800e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:12.098037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fec21eb8-4654-4eb0-8a12-27c3361d800e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:12.135514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:12.193723 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 734/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:41:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:12.231593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd91a1ff-f8e2-42d2-9223-0015c3d4ee31 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:12.233192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-dd91a1ff-f8e2-42d2-9223-0015c3d4ee31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:12.337922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd91a1ff-f8e2-42d2-9223-0015c3d4ee31 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:12.375605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:12.411281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22436f3-ea7a-45f5-8bed-1d14eab6f4d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:12.411910 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22436f3-ea7a-45f5-8bed-1d14eab6f4d2 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:12.412196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22436f3-ea7a-45f5-8bed-1d14eab6f4d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:12.412469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22436f3-ea7a-45f5-8bed-1d14eab6f4d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:12.414129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d22436f3-ea7a-45f5-8bed-1d14eab6f4d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:12.520999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22436f3-ea7a-45f5-8bed-1d14eab6f4d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:12.522432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22436f3-ea7a-45f5-8bed-1d14eab6f4d2 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:12.540132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 730/1464] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:14.562246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91217f9-758c-4b55-bf1c-9144e493766d None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:14.562812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91217f9-758c-4b55-bf1c-9144e493766d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:14.562812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91217f9-758c-4b55-bf1c-9144e493766d None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:14.564159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 735/1465] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:41:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:14.569386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6231fbf-61e6-4b41-90c7-1730de85bd6a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:14.569838 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6231fbf-61e6-4b41-90c7-1730de85bd6a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:14.569940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6231fbf-61e6-4b41-90c7-1730de85bd6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:14.634536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-e6231fbf-61e6-4b41-90c7-1730de85bd6a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:14.636017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e6231fbf-61e6-4b41-90c7-1730de85bd6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:14.654150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:14.732647 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 731/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:41:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:14.769320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91217f9-758c-4b55-bf1c-9144e493766d None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:14.770895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f91217f9-758c-4b55-bf1c-9144e493766d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:14.877056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91217f9-758c-4b55-bf1c-9144e493766d None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:14.945903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:14.954713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575ef422-3de2-455a-b3f9-32a5158bd896 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:14.955347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575ef422-3de2-455a-b3f9-32a5158bd896 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:14.955634 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575ef422-3de2-455a-b3f9-32a5158bd896 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:14.955915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575ef422-3de2-455a-b3f9-32a5158bd896 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:14.957722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-575ef422-3de2-455a-b3f9-32a5158bd896 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:15.066842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575ef422-3de2-455a-b3f9-32a5158bd896 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:15.068292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575ef422-3de2-455a-b3f9-32a5158bd896 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:15.086719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 736/1467] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:17.085853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588bcde1-3f76-4dd8-8880-193426bcd63a None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:17.086421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588bcde1-3f76-4dd8-8880-193426bcd63a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:17.086421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588bcde1-3f76-4dd8-8880-193426bcd63a None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:17.087762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 732/1468] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:41:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:17.093772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3893f3c-c675-4100-9f90-fe04f77a100b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:17.094160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3893f3c-c675-4100-9f90-fe04f77a100b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:17.094344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3893f3c-c675-4100-9f90-fe04f77a100b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:17.160198 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f3893f3c-c675-4100-9f90-fe04f77a100b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:17.161444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f3893f3c-c675-4100-9f90-fe04f77a100b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:17.209831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:17.257686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 737/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:41:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:17.294779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588bcde1-3f76-4dd8-8880-193426bcd63a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:17.296800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-588bcde1-3f76-4dd8-8880-193426bcd63a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:17.401654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588bcde1-3f76-4dd8-8880-193426bcd63a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:17.449528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:17.476737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40204f2-9ac3-438c-9fe3-e56bda14f81a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:17.477377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40204f2-9ac3-438c-9fe3-e56bda14f81a service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:17.477656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40204f2-9ac3-438c-9fe3-e56bda14f81a service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:17.477930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40204f2-9ac3-438c-9fe3-e56bda14f81a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:17.479541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a40204f2-9ac3-438c-9fe3-e56bda14f81a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:17.590587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40204f2-9ac3-438c-9fe3-e56bda14f81a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:17.592094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40204f2-9ac3-438c-9fe3-e56bda14f81a service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:17.610311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 733/1470] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:27.255849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=22 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:27.608420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=66 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:37.130229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e4e928-fe66-4cd1-82e9-aa7d0ce10601 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:37.130755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e4e928-fe66-4cd1-82e9-aa7d0ce10601 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:37.130755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e4e928-fe66-4cd1-82e9-aa7d0ce10601 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:37.132125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 738/1471] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:41:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:37.137495 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdff538c-4bb4-49ef-9ceb-f667ec88fecd None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:37.137915 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdff538c-4bb4-49ef-9ceb-f667ec88fecd None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:37.138062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdff538c-4bb4-49ef-9ceb-f667ec88fecd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:37.208447 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-cdff538c-4bb4-49ef-9ceb-f667ec88fecd None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:37.209714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-cdff538c-4bb4-49ef-9ceb-f667ec88fecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:37.309693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 734/1472] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:41:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:37.396826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6be46-6832-47b2-a95c-dce0ab66be87 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:37.397117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6be46-6832-47b2-a95c-dce0ab66be87 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:37.397400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6be46-6832-47b2-a95c-dce0ab66be87 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:37.398769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 739/1473] 10.222.0.40 () {54 vars in 966 bytes} [Fri Jul 29 04:41:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:37.403816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb4b738-bc04-413b-afbf-e9e79c3b21cb None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:37.404113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb4b738-bc04-413b-afbf-e9e79c3b21cb None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:37.404451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb4b738-bc04-413b-afbf-e9e79c3b21cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:37.481529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9fb4b738-bc04-413b-afbf-e9e79c3b21cb None None] MFA Rules not processed for user `3749dc021aac478f88072ced48f4cee4`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:37.482857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9fb4b738-bc04-413b-afbf-e9e79c3b21cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:37.503726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:37.563511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 735/1474] 10.222.0.40 () {58 vars in 1063 bytes} [Fri Jul 29 04:41:37 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:37.571062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e6be46-6832-47b2-a95c-dce0ab66be87 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:37.572573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d0e6be46-6832-47b2-a95c-dce0ab66be87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:37.675801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e6be46-6832-47b2-a95c-dce0ab66be87 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:37.748683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e07a674-c0c1-4e2e-931a-8dc1b91cb836 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3749dc021aac478f88072ced48f4cee4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:37.749302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e07a674-c0c1-4e2e-931a-8dc1b91cb836 service glance] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:37.749587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e07a674-c0c1-4e2e-931a-8dc1b91cb836 service glance] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:37.749863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e07a674-c0c1-4e2e-931a-8dc1b91cb836 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:37.751487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9e07a674-c0c1-4e2e-931a-8dc1b91cb836 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:37.805518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:37.860537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e07a674-c0c1-4e2e-931a-8dc1b91cb836 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:37.862018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e07a674-c0c1-4e2e-931a-8dc1b91cb836 service glance] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:37.882462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 740/1475] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:39.979430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff9af20-5fd5-4c14-8105-144ffddfa8f3 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:39.979963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff9af20-5fd5-4c14-8105-144ffddfa8f3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:39.979963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff9af20-5fd5-4c14-8105-144ffddfa8f3 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:39.981325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 736/1476] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:41:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:39.987453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a38178-bb17-4b84-8871-87e8e0641713 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:39.987920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a38178-bb17-4b84-8871-87e8e0641713 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:39.988027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a38178-bb17-4b84-8871-87e8e0641713 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:40.052681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-56a38178-bb17-4b84-8871-87e8e0641713 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:40.053884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-56a38178-bb17-4b84-8871-87e8e0641713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:40.148559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 741/1477] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:41:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:40.172089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff9af20-5fd5-4c14-8105-144ffddfa8f3 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:40.173656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8ff9af20-5fd5-4c14-8105-144ffddfa8f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:40.299536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff9af20-5fd5-4c14-8105-144ffddfa8f3 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:40.348335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:40.393718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39ed433-c98f-4ba3-89b6-be988dde8e6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:40.394336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39ed433-c98f-4ba3-89b6-be988dde8e6c admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:40.394623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39ed433-c98f-4ba3-89b6-be988dde8e6c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:40.394900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39ed433-c98f-4ba3-89b6-be988dde8e6c admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:40.395603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39ed433-c98f-4ba3-89b6-be988dde8e6c admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:40.397384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WARNING py.warnings [None req-c39ed433-c98f-4ba3-89b6-be988dde8e6c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:41:40.397384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: warnings.warn(msg) Jul 29 04:41:40.397384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: Jul 29 04:41:40.398646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39ed433-c98f-4ba3-89b6-be988dde8e6c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:40.417358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 737/1478] 10.222.0.40 () {56 vars in 1108 bytes} [Fri Jul 29 04:41:40 2022] GET /identity/v3/projects => generated 2553 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:41:42.416040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f30345-eaed-40a8-a45c-9d40a4e60dfc None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:42.416656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f30345-eaed-40a8-a45c-9d40a4e60dfc None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:42.416656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f30345-eaed-40a8-a45c-9d40a4e60dfc None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:42.417961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 742/1479] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:41:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:42.423971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899bd397-5230-492f-83d6-6b850c85d23b None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:42.424431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899bd397-5230-492f-83d6-6b850c85d23b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:42.424642 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899bd397-5230-492f-83d6-6b850c85d23b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:42.489639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-899bd397-5230-492f-83d6-6b850c85d23b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:42.490858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-899bd397-5230-492f-83d6-6b850c85d23b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:42.585875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 738/1480] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:41:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:41:42.629288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f30345-eaed-40a8-a45c-9d40a4e60dfc None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:42.630751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-13f30345-eaed-40a8-a45c-9d40a4e60dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:42.640386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:42.777166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f30345-eaed-40a8-a45c-9d40a4e60dfc None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:42.859460 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:42.892393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b79c5ae-f66e-484c-a156-bb9380e4d540 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:42.893043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b79c5ae-f66e-484c-a156-bb9380e4d540 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:42.893331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b79c5ae-f66e-484c-a156-bb9380e4d540 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:42.893608 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b79c5ae-f66e-484c-a156-bb9380e4d540 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:42.895278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8b79c5ae-f66e-484c-a156-bb9380e4d540 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:42.984198 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b79c5ae-f66e-484c-a156-bb9380e4d540 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:42.985844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b79c5ae-f66e-484c-a156-bb9380e4d540 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:43.003773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 743/1481] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:45.191738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935ee2f-d3f2-4dbd-992b-90d87e16ab89 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:45.192288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935ee2f-d3f2-4dbd-992b-90d87e16ab89 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:45.192288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935ee2f-d3f2-4dbd-992b-90d87e16ab89 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:45.193713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 739/1482] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:41:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:45.198952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9e0214-9649-4162-9968-b7b8ab470620 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:45.199401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9e0214-9649-4162-9968-b7b8ab470620 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:45.199498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9e0214-9649-4162-9968-b7b8ab470620 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:45.265484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-ee9e0214-9649-4162-9968-b7b8ab470620 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:45.266702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ee9e0214-9649-4162-9968-b7b8ab470620 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:45.313658 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:45.367566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 744/1483] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:41:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:45.405626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7935ee2f-d3f2-4dbd-992b-90d87e16ab89 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:45.407215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7935ee2f-d3f2-4dbd-992b-90d87e16ab89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:45.431821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:45.516120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7935ee2f-d3f2-4dbd-992b-90d87e16ab89 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:45.591260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03114919-1113-425e-b1aa-aeaf6c707460 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:45.592473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03114919-1113-425e-b1aa-aeaf6c707460 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:45.592993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03114919-1113-425e-b1aa-aeaf6c707460 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:45.593284 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03114919-1113-425e-b1aa-aeaf6c707460 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:45.594937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-03114919-1113-425e-b1aa-aeaf6c707460 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:45.670136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:45.707512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03114919-1113-425e-b1aa-aeaf6c707460 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:45.709001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03114919-1113-425e-b1aa-aeaf6c707460 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:45.726972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 740/1484] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:47.787748 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b38d02f-acef-4fa1-b019-e0429a5f83c8 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:47.788279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b38d02f-acef-4fa1-b019-e0429a5f83c8 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:47.788404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b38d02f-acef-4fa1-b019-e0429a5f83c8 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:47.789747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 745/1485] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:41:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:47.796021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7428c111-a596-4530-8e51-5d75b8a25fac None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:47.796541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7428c111-a596-4530-8e51-5d75b8a25fac None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:47.796672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7428c111-a596-4530-8e51-5d75b8a25fac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:47.862534 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7428c111-a596-4530-8e51-5d75b8a25fac None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:47.863766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7428c111-a596-4530-8e51-5d75b8a25fac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:47.962888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 741/1486] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:41:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:48.002000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b38d02f-acef-4fa1-b019-e0429a5f83c8 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:48.003547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3b38d02f-acef-4fa1-b019-e0429a5f83c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:48.106437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b38d02f-acef-4fa1-b019-e0429a5f83c8 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:48.178475 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:48.180156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc9664-1798-42b3-8e48-4c65a2b6d401 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:48.180821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc9664-1798-42b3-8e48-4c65a2b6d401 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:48.181291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc9664-1798-42b3-8e48-4c65a2b6d401 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:48.181615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc9664-1798-42b3-8e48-4c65a2b6d401 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:48.183579 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d2cc9664-1798-42b3-8e48-4c65a2b6d401 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:48.291453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cc9664-1798-42b3-8e48-4c65a2b6d401 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:48.292938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cc9664-1798-42b3-8e48-4c65a2b6d401 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:48.311635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 746/1487] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:41:50.404973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44837a5e-cf41-484b-8e68-48db006a4dfb None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:50.405497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44837a5e-cf41-484b-8e68-48db006a4dfb None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:50.405497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44837a5e-cf41-484b-8e68-48db006a4dfb None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:50.406856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 742/1488] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:41:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:50.412266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23721f19-31ff-4361-95b0-d44f5a505441 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:50.412719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23721f19-31ff-4361-95b0-d44f5a505441 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:50.412872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23721f19-31ff-4361-95b0-d44f5a505441 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:50.479140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-23721f19-31ff-4361-95b0-d44f5a505441 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:50.480358 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-23721f19-31ff-4361-95b0-d44f5a505441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:50.532626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:50.577462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 747/1489] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:41:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:50.619181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44837a5e-cf41-484b-8e68-48db006a4dfb None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:50.620796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-44837a5e-cf41-484b-8e68-48db006a4dfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:50.658820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:50.728692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44837a5e-cf41-484b-8e68-48db006a4dfb None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:50.801769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5d7f9b-0955-450c-b233-86f6e872940f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:50.802398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5d7f9b-0955-450c-b233-86f6e872940f service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:50.802689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5d7f9b-0955-450c-b233-86f6e872940f service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:50.802974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5d7f9b-0955-450c-b233-86f6e872940f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:50.804649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0d5d7f9b-0955-450c-b233-86f6e872940f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:50.884222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:50.915355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5d7f9b-0955-450c-b233-86f6e872940f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:50.916865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5d7f9b-0955-450c-b233-86f6e872940f service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:50.935393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 743/1490] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:41:53.150172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c01038-2136-4008-a827-a991fc3cf6ba None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:53.150660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c01038-2136-4008-a827-a991fc3cf6ba None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:53.150754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c01038-2136-4008-a827-a991fc3cf6ba None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:53.152116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 748/1491] 10.222.0.40 () {54 vars in 926 bytes} [Fri Jul 29 04:41:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:41:53.157414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dc3b28-c07b-48db-89f3-f2425df5ec08 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:53.157862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dc3b28-c07b-48db-89f3-f2425df5ec08 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:53.157977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0dc3b28-c07b-48db-89f3-f2425df5ec08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:53.224774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d0dc3b28-c07b-48db-89f3-f2425df5ec08 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:41:53.226014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d0dc3b28-c07b-48db-89f3-f2425df5ec08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:53.328755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 744/1492] 10.222.0.40 () {58 vars in 1023 bytes} [Fri Jul 29 04:41:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:41:53.368740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c01038-2136-4008-a827-a991fc3cf6ba None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:41:53.370291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-04c01038-2136-4008-a827-a991fc3cf6ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:53.474052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c01038-2136-4008-a827-a991fc3cf6ba None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:41:53.549415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1635b2f-5c44-4697-b723-e1d453d101e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:41:53.550083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1635b2f-5c44-4697-b723-e1d453d101e1 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:41:53.550371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1635b2f-5c44-4697-b723-e1d453d101e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:41:53.550647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1635b2f-5c44-4697-b723-e1d453d101e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:41:53.552296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c1635b2f-5c44-4697-b723-e1d453d101e1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:41:53.561638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:41:53.662492 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1635b2f-5c44-4697-b723-e1d453d101e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:41:53.663949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1635b2f-5c44-4697-b723-e1d453d101e1 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:41:53.682390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 749/1493] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:41:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:42:03.326928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=82 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:42:03.680083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=50 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:25.602473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7235c4ad-5699-4621-9cb0-4cdd3a329046 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:25.603065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7235c4ad-5699-4621-9cb0-4cdd3a329046 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:25.603065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7235c4ad-5699-4621-9cb0-4cdd3a329046 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:25.679821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7235c4ad-5699-4621-9cb0-4cdd3a329046 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:43:25.681097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7235c4ad-5699-4621-9cb0-4cdd3a329046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:25.777605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 745/1494] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:43:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:43:25.785900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfbd39a-6c8b-475f-a4d5-9bcba8922cb7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:25.786413 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfbd39a-6c8b-475f-a4d5-9bcba8922cb7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:25.786513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dfbd39a-6c8b-475f-a4d5-9bcba8922cb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:25.862511 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0dfbd39a-6c8b-475f-a4d5-9bcba8922cb7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:43:25.863636 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0dfbd39a-6c8b-475f-a4d5-9bcba8922cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:25.967575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 750/1495] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:43:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:43:25.972241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7235c4ad-5699-4621-9cb0-4cdd3a329046 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:43:25.973494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7235c4ad-5699-4621-9cb0-4cdd3a329046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:25.985039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7235c4ad-5699-4621-9cb0-4cdd3a329046 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:43:26.045666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:26.081450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:43:26.082089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:26.082371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:26.082643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:26.083331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:43:26.083634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:43:26.084656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WARNING py.warnings [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:43:26.084656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: warnings.warn(msg) Jul 29 04:43:26.084656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: Jul 29 04:43:26.085541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:43:26.097174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 746/1496] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:43:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:43:26.105729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfbd39a-6c8b-475f-a4d5-9bcba8922cb7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:43:26.114757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dfbd39a-6c8b-475f-a4d5-9bcba8922cb7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:43:26.176854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:26.211078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab35e0b-a7cf-4255-b0f2-e2f05fe522b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:43:26.211693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab35e0b-a7cf-4255-b0f2-e2f05fe522b1 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:26.211974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab35e0b-a7cf-4255-b0f2-e2f05fe522b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:26.212247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab35e0b-a7cf-4255-b0f2-e2f05fe522b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:26.212956 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab35e0b-a7cf-4255-b0f2-e2f05fe522b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:43:26.214697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: WARNING py.warnings [None req-eab35e0b-a7cf-4255-b0f2-e2f05fe522b1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:43:26.214697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: warnings.warn(msg) Jul 29 04:43:26.214697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: Jul 29 04:43:26.216035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab35e0b-a7cf-4255-b0f2-e2f05fe522b1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:43:26.272394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 751/1497] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:43:26 2022] POST /identity/v3/projects => generated 377 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:43:26.276652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:43:26.278289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:26.316415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ea2d3f-4748-4c84-85f5-70623f845bff admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:43:26.411876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16401928-20bd-4716-a0cd-5693aa9ae424 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:43:26.412420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16401928-20bd-4716-a0cd-5693aa9ae424 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:26.412742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16401928-20bd-4716-a0cd-5693aa9ae424 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:26.413026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16401928-20bd-4716-a0cd-5693aa9ae424 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:26.413703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16401928-20bd-4716-a0cd-5693aa9ae424 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:43:26.414716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WARNING py.warnings [None req-16401928-20bd-4716-a0cd-5693aa9ae424 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:43:26.414716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: warnings.warn(msg) Jul 29 04:43:26.414716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: Jul 29 04:43:26.415931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16401928-20bd-4716-a0cd-5693aa9ae424 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:43:26.458168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 747/1498] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:43:26 2022] POST /identity/v3/users => generated 289 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:43:26.462376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab35e0b-a7cf-4255-b0f2-e2f05fe522b1 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:43:26.463944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-eab35e0b-a7cf-4255-b0f2-e2f05fe522b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:26.526156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab35e0b-a7cf-4255-b0f2-e2f05fe522b1 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:43:26.609154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:26.619904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ecd17a-048a-434e-98d0-40a59af18701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:43:26.620567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ecd17a-048a-434e-98d0-40a59af18701 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:26.621704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ecd17a-048a-434e-98d0-40a59af18701 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:26.621704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ecd17a-048a-434e-98d0-40a59af18701 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:26.622330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ecd17a-048a-434e-98d0-40a59af18701 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:43:26.623348 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: WARNING py.warnings [None req-b3ecd17a-048a-434e-98d0-40a59af18701 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:43:26.623348 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: warnings.warn(msg) Jul 29 04:43:26.623348 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: Jul 29 04:43:26.624249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ecd17a-048a-434e-98d0-40a59af18701 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:43:26.636580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 752/1499] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:43:26 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:43:26.640785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16401928-20bd-4716-a0cd-5693aa9ae424 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:43:26.642342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-16401928-20bd-4716-a0cd-5693aa9ae424 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:26.654092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16401928-20bd-4716-a0cd-5693aa9ae424 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:43:26.680157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:26.749263 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70b2477-2731-43ce-9d0a-10e75a6b0ec8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:43:26.749880 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70b2477-2731-43ce-9d0a-10e75a6b0ec8 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:26.750160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70b2477-2731-43ce-9d0a-10e75a6b0ec8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:26.750428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70b2477-2731-43ce-9d0a-10e75a6b0ec8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:26.751119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70b2477-2731-43ce-9d0a-10e75a6b0ec8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:43:26.752125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WARNING py.warnings [None req-e70b2477-2731-43ce-9d0a-10e75a6b0ec8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:43:26.752125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: warnings.warn(msg) Jul 29 04:43:26.752125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: Jul 29 04:43:26.753037 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70b2477-2731-43ce-9d0a-10e75a6b0ec8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:43:26.765186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 748/1500] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:43:26 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:43:26.769550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ecd17a-048a-434e-98d0-40a59af18701 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:43:26.771100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b3ecd17a-048a-434e-98d0-40a59af18701 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:26.804898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ecd17a-048a-434e-98d0-40a59af18701 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:43:26.898622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fafd06-c3e5-480c-9728-5b73f39b00c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:43:26.899303 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fafd06-c3e5-480c-9728-5b73f39b00c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:26.899592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fafd06-c3e5-480c-9728-5b73f39b00c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:26.899888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fafd06-c3e5-480c-9728-5b73f39b00c0 admin admin] PATH_INFO: `/v3/projects/79c76fe60bb34dd6aade72fc5e96a195/users/40dd170e4e404959bd49fd795b695fe7/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:26.922349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fafd06-c3e5-480c-9728-5b73f39b00c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=79c76fe60bb34dd6aade72fc5e96a195, user_id=40dd170e4e404959bd49fd795b695fe7, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:43:26.923381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: WARNING py.warnings [None req-87fafd06-c3e5-480c-9728-5b73f39b00c0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:43:26.923381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: warnings.warn(msg) Jul 29 04:43:26.923381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: Jul 29 04:43:26.925379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fafd06-c3e5-480c-9728-5b73f39b00c0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:43:26.950205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 753/1501] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:43:26 2022] PUT /identity/v3/projects/79c76fe60bb34dd6aade72fc5e96a195/users/40dd170e4e404959bd49fd795b695fe7/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:43:26.962367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b3d221-2e4e-46dc-874f-511c9cbca8ea None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:26.962664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b3d221-2e4e-46dc-874f-511c9cbca8ea None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:26.962946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b3d221-2e4e-46dc-874f-511c9cbca8ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:27.019810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-08b3d221-2e4e-46dc-874f-511c9cbca8ea None None] MFA Rules not processed for user `40dd170e4e404959bd49fd795b695fe7`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:43:27.021074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-08b3d221-2e4e-46dc-874f-511c9cbca8ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:27.099420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 749/1502] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:43:26 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:43:27.112891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08368795-d63c-44b5-98c9-e98d3a02d4a8 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:27.113186 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08368795-d63c-44b5-98c9-e98d3a02d4a8 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:27.113473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08368795-d63c-44b5-98c9-e98d3a02d4a8 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:27.114818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 754/1503] 10.222.0.40 () {54 vars in 964 bytes} [Fri Jul 29 04:43:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:43:27.119731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc452d9-ddd3-44ce-b0c6-dbd0a90f62ce None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:27.120049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc452d9-ddd3-44ce-b0c6-dbd0a90f62ce None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:27.120355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc452d9-ddd3-44ce-b0c6-dbd0a90f62ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:27.150977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:27.197249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-ccc452d9-ddd3-44ce-b0c6-dbd0a90f62ce None None] MFA Rules not processed for user `de49172e2a054096b319a7e4349ecaea`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:43:27.198471 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ccc452d9-ddd3-44ce-b0c6-dbd0a90f62ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:27.273418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 750/1504] 10.222.0.40 () {58 vars in 1061 bytes} [Fri Jul 29 04:43:27 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:43:27.281160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08368795-d63c-44b5-98c9-e98d3a02d4a8 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:43:27.282883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-08368795-d63c-44b5-98c9-e98d3a02d4a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:27.333589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:27.388752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08368795-d63c-44b5-98c9-e98d3a02d4a8 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:43:27.461423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc31eb9-35cb-46b4-a74c-eba0ff3e1968 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:43:27.462047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc31eb9-35cb-46b4-a74c-eba0ff3e1968 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:27.462333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc31eb9-35cb-46b4-a74c-eba0ff3e1968 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:27.462613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc31eb9-35cb-46b4-a74c-eba0ff3e1968 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:27.464358 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7bc31eb9-35cb-46b4-a74c-eba0ff3e1968 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:27.567826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc31eb9-35cb-46b4-a74c-eba0ff3e1968 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:43:27.569299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc31eb9-35cb-46b4-a74c-eba0ff3e1968 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:43:27.572799 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:27.587656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 755/1505] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:43:27 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:43:27.635256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafff6df-ec09-4690-ae75-997c6d806fab None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:27.635547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafff6df-ec09-4690-ae75-997c6d806fab None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:27.635826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafff6df-ec09-4690-ae75-997c6d806fab None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:27.637300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 751/1506] 10.222.0.40 () {52 vars in 830 bytes} [Fri Jul 29 04:43:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:43:27.640863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc31eb9-35cb-46b4-a74c-eba0ff3e1968 service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:43:27.642117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7bc31eb9-35cb-46b4-a74c-eba0ff3e1968 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:27.743293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc31eb9-35cb-46b4-a74c-eba0ff3e1968 service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:43:27.836592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:43:27.837241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:27.837483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:27.837762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] PATH_INFO: `/v3/users/40dd170e4e404959bd49fd795b695fe7` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:27.851703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] RBAC: Authorizing `identity:delete_user(user_id=40dd170e4e404959bd49fd795b695fe7)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:43:27.852771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: WARNING py.warnings [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:43:27.852771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: warnings.warn(msg) Jul 29 04:43:27.852771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: Jul 29 04:43:27.853984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:43:27.863727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:27.917491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40dd170e4e404959bd49fd795b695fe7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:43:27.926940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40dd170e4e404959bd49fd795b695fe7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:43:27.927467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40dd170e4e404959bd49fd795b695fe7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:43:27.927900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40dd170e4e404959bd49fd795b695fe7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:43:27.940256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40dd170e4e404959bd49fd795b695fe7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:43:27.965620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 756/1507] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:43:27 2022] DELETE /identity/v3/users/40dd170e4e404959bd49fd795b695fe7 => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:43:27.970027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fb2bff-b024-4d03-8eab-66c22d14713d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:27.970504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fb2bff-b024-4d03-8eab-66c22d14713d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:27.970624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fb2bff-b024-4d03-8eab-66c22d14713d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:28.027517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c2fb2bff-b024-4d03-8eab-66c22d14713d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:43:28.028773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c2fb2bff-b024-4d03-8eab-66c22d14713d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:28.104063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:28.117717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 752/1508] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:43:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:43:28.128185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:43:28.129482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:28.230953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e5766a-b656-488d-a7f5-f64d1671c114 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:43:28.286274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:28.307745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47027c2-08f1-48d3-8f4b-8f75984c3a37 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:43:28.308947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47027c2-08f1-48d3-8f4b-8f75984c3a37 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:28.309228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47027c2-08f1-48d3-8f4b-8f75984c3a37 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:28.309502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47027c2-08f1-48d3-8f4b-8f75984c3a37 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:28.311114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e47027c2-08f1-48d3-8f4b-8f75984c3a37 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:28.419256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47027c2-08f1-48d3-8f4b-8f75984c3a37 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:43:28.420634 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47027c2-08f1-48d3-8f4b-8f75984c3a37 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:43:28.440288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 757/1509] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:43:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:43:28.760419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fb2bff-b024-4d03-8eab-66c22d14713d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:43:28.761831 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c2fb2bff-b024-4d03-8eab-66c22d14713d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:43:28.773368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fb2bff-b024-4d03-8eab-66c22d14713d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:43:28.873128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:43:28.873776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:43:28.874065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:43:28.874345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] PATH_INFO: `/v3/projects/79c76fe60bb34dd6aade72fc5e96a195` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:43:28.884109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] RBAC: Authorizing `identity:delete_project(project_id=79c76fe60bb34dd6aade72fc5e96a195)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:43:28.885434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WARNING py.warnings [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:43:28.885434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: warnings.warn(msg) Jul 29 04:43:28.885434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: Jul 29 04:43:28.886688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:43:28.944568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79c76fe60bb34dd6aade72fc5e96a195', 'request_id': 'req-b89cf194-80f6-46a7-a4c5-82d203eaf696'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:43:28.957495 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79c76fe60bb34dd6aade72fc5e96a195', 'request_id': 'req-b89cf194-80f6-46a7-a4c5-82d203eaf696'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:43:28.966614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79c76fe60bb34dd6aade72fc5e96a195', 'request_id': 'req-b89cf194-80f6-46a7-a4c5-82d203eaf696'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:43:28.991344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] The token cache is being invalidate because project 79c76fe60bb34dd6aade72fc5e96a195 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:43:28.991986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:43:28.992084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b89cf194-80f6-46a7-a4c5-82d203eaf696 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:43:28.993494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 753/1510] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:43:28 2022] DELETE /identity/v3/projects/79c76fe60bb34dd6aade72fc5e96a195 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:43:38.438003 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=62 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:43:38.989335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:45:14.092491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bac6d7b-cd6c-4ebe-88c4-6b6b2f470248 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:14.093132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bac6d7b-cd6c-4ebe-88c4-6b6b2f470248 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:14.093132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bac6d7b-cd6c-4ebe-88c4-6b6b2f470248 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:14.094525 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 758/1511] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:45:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:45:14.099276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbf7870-ad96-480a-a26b-aca3df5fdfb6 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:14.099679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbf7870-ad96-480a-a26b-aca3df5fdfb6 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:14.099837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbf7870-ad96-480a-a26b-aca3df5fdfb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:14.182403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-dfbf7870-ad96-480a-a26b-aca3df5fdfb6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:45:14.183641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-dfbf7870-ad96-480a-a26b-aca3df5fdfb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:14.287064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 754/1512] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:45:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:45:14.442650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bac6d7b-cd6c-4ebe-88c4-6b6b2f470248 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:45:14.444812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2bac6d7b-cd6c-4ebe-88c4-6b6b2f470248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:14.564354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bac6d7b-cd6c-4ebe-88c4-6b6b2f470248 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:45:14.569871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789a9972-d53c-4f28-bfe4-dfaec174040f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:45:14.570496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789a9972-d53c-4f28-bfe4-dfaec174040f service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:14.570784 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789a9972-d53c-4f28-bfe4-dfaec174040f service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:14.571063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789a9972-d53c-4f28-bfe4-dfaec174040f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:14.579126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789a9972-d53c-4f28-bfe4-dfaec174040f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:45:14.580740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789a9972-d53c-4f28-bfe4-dfaec174040f service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:45:14.590082 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 759/1513] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:45:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:45:15.419926 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7389ffc6-fb29-4632-87a2-2f88f5fcdea8 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:15.420504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7389ffc6-fb29-4632-87a2-2f88f5fcdea8 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:15.420504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7389ffc6-fb29-4632-87a2-2f88f5fcdea8 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:15.421894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 755/1514] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:45:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:45:15.426994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31896ecd-8a6c-4a7d-b991-fafda295cafa None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:15.427475 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31896ecd-8a6c-4a7d-b991-fafda295cafa None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:15.427584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31896ecd-8a6c-4a7d-b991-fafda295cafa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:15.448403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-31896ecd-8a6c-4a7d-b991-fafda295cafa None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:45:15.449656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-31896ecd-8a6c-4a7d-b991-fafda295cafa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:15.457790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 760/1515] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:45:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:45:15.615211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7389ffc6-fb29-4632-87a2-2f88f5fcdea8 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:45:15.630204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7389ffc6-fb29-4632-87a2-2f88f5fcdea8 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:45:15.635632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a07642-6550-43f1-aed5-2ea959b0d266 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:45:15.636243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a07642-6550-43f1-aed5-2ea959b0d266 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:15.636527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a07642-6550-43f1-aed5-2ea959b0d266 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:15.636833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a07642-6550-43f1-aed5-2ea959b0d266 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:15.644788 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a07642-6550-43f1-aed5-2ea959b0d266 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:45:15.646371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a07642-6550-43f1-aed5-2ea959b0d266 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:45:15.656710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 756/1516] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:45:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:45:15.921088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4695636e-a96a-4968-9bf9-369fbe497942 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:15.921333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4695636e-a96a-4968-9bf9-369fbe497942 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:15.921616 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4695636e-a96a-4968-9bf9-369fbe497942 None None] PATH_INFO: `/v3` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:15.923171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 761/1517] 10.222.0.64 () {54 vars in 925 bytes} [Fri Jul 29 04:45:15 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:45:15.929557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be41746-1d37-43dd-a74e-291d66bc269e None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:15.929856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be41746-1d37-43dd-a74e-291d66bc269e None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:15.930144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be41746-1d37-43dd-a74e-291d66bc269e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:15.989301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-8be41746-1d37-43dd-a74e-291d66bc269e None None] MFA Rules not processed for user `e3da5e428049419a9b35494655f52484`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:45:15.990508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8be41746-1d37-43dd-a74e-291d66bc269e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:16.007494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:45:16.061963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 757/1518] 10.222.0.64 () {58 vars in 1015 bytes} [Fri Jul 29 04:45:15 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:45:25.348855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19db6f21-3f04-48f3-b218-8617e050fd25 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:25.349379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19db6f21-3f04-48f3-b218-8617e050fd25 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:25.349379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19db6f21-3f04-48f3-b218-8617e050fd25 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:25.350765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 762/1519] 10.222.0.40 () {54 vars in 961 bytes} [Fri Jul 29 04:45:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:45:25.355822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c4d94d-019f-498a-acec-d8f5b98b3706 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:25.356273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c4d94d-019f-498a-acec-d8f5b98b3706 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:25.356394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c4d94d-019f-498a-acec-d8f5b98b3706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:25.375798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-35c4d94d-019f-498a-acec-d8f5b98b3706 None None] MFA Rules not processed for user `e3da5e428049419a9b35494655f52484`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:45:25.377056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-35c4d94d-019f-498a-acec-d8f5b98b3706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:25.386223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 758/1520] 10.222.0.40 () {58 vars in 1058 bytes} [Fri Jul 29 04:45:25 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:45:25.393754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19db6f21-3f04-48f3-b218-8617e050fd25 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:45:25.407589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19db6f21-3f04-48f3-b218-8617e050fd25 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:45:25.413640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30bb5a7a-213d-41c2-abad-c72a66b0bed3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:45:25.414255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb5a7a-213d-41c2-abad-c72a66b0bed3 service placement] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:25.414538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb5a7a-213d-41c2-abad-c72a66b0bed3 service placement] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:25.414886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bb5a7a-213d-41c2-abad-c72a66b0bed3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:25.423741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bb5a7a-213d-41c2-abad-c72a66b0bed3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:45:25.425371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30bb5a7a-213d-41c2-abad-c72a66b0bed3 service placement] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:45:25.435437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 763/1521] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:45:25 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:45:26.878819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6092a-ea3d-4aad-910c-34b4b4eff814 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:26.879377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6092a-ea3d-4aad-910c-34b4b4eff814 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:26.879377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6092a-ea3d-4aad-910c-34b4b4eff814 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:26.880770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 759/1522] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:45:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:45:26.885692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61dd26f-a505-4ae2-bb56-432c5a83339b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:26.886155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61dd26f-a505-4ae2-bb56-432c5a83339b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:26.886267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61dd26f-a505-4ae2-bb56-432c5a83339b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:26.906416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e61dd26f-a505-4ae2-bb56-432c5a83339b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:45:26.907543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e61dd26f-a505-4ae2-bb56-432c5a83339b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:26.916452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 764/1523] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:45:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:45:27.072036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec6092a-ea3d-4aad-910c-34b4b4eff814 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:45:27.086231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec6092a-ea3d-4aad-910c-34b4b4eff814 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:45:27.092659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a63bf1-07d6-46aa-bdf2-ddd0e58d23be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:45:27.093394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a63bf1-07d6-46aa-bdf2-ddd0e58d23be service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:27.093679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a63bf1-07d6-46aa-bdf2-ddd0e58d23be service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:27.093956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a63bf1-07d6-46aa-bdf2-ddd0e58d23be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:27.103399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a63bf1-07d6-46aa-bdf2-ddd0e58d23be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:45:27.105135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a63bf1-07d6-46aa-bdf2-ddd0e58d23be service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:45:27.115137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 760/1524] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:45:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:45:27.941489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b423b-9f66-4ac8-aa51-9a2d716554db None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:27.942009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b423b-9f66-4ac8-aa51-9a2d716554db None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:27.942009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785b423b-9f66-4ac8-aa51-9a2d716554db None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:27.943329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 765/1525] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jul 29 04:45:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:45:27.948243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe04f742-273d-42d8-85b4-72d9d73f551b None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:27.948709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe04f742-273d-42d8-85b4-72d9d73f551b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:27.948846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe04f742-273d-42d8-85b4-72d9d73f551b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:27.968406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-fe04f742-273d-42d8-85b4-72d9d73f551b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:45:27.969659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fe04f742-273d-42d8-85b4-72d9d73f551b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:27.978175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 761/1526] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jul 29 04:45:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:45:28.133100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b423b-9f66-4ac8-aa51-9a2d716554db None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:45:28.148009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785b423b-9f66-4ac8-aa51-9a2d716554db None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:45:28.154216 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443c2bd7-b77a-4be3-8ae5-b938a59aea32 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:45:28.154830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443c2bd7-b77a-4be3-8ae5-b938a59aea32 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:28.155121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443c2bd7-b77a-4be3-8ae5-b938a59aea32 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:28.155404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443c2bd7-b77a-4be3-8ae5-b938a59aea32 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:28.163567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443c2bd7-b77a-4be3-8ae5-b938a59aea32 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:45:28.165318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443c2bd7-b77a-4be3-8ae5-b938a59aea32 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:45:28.175145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 766/1527] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:45:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:45:31.934656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d1ddff-e5f2-4e88-99a0-b36cfdcc55cd None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:31.935193 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d1ddff-e5f2-4e88-99a0-b36cfdcc55cd None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:31.935193 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d1ddff-e5f2-4e88-99a0-b36cfdcc55cd None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:31.936573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 762/1528] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:45:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:45:31.941620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3153a7e4-372a-41b4-8e37-51b7e3b6cc47 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:31.942063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3153a7e4-372a-41b4-8e37-51b7e3b6cc47 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:31.942169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3153a7e4-372a-41b4-8e37-51b7e3b6cc47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:31.961642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-3153a7e4-372a-41b4-8e37-51b7e3b6cc47 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:45:31.962835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3153a7e4-372a-41b4-8e37-51b7e3b6cc47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:31.971496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 767/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:45:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:45:32.044693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d1ddff-e5f2-4e88-99a0-b36cfdcc55cd None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:45:32.046426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-25d1ddff-e5f2-4e88-99a0-b36cfdcc55cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:32.125626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d1ddff-e5f2-4e88-99a0-b36cfdcc55cd None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:45:32.129444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb3e8c3-eed7-468f-836b-85805c7d1b52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:45:32.130066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb3e8c3-eed7-468f-836b-85805c7d1b52 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:32.130355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb3e8c3-eed7-468f-836b-85805c7d1b52 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:32.130665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb3e8c3-eed7-468f-836b-85805c7d1b52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:32.139853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb3e8c3-eed7-468f-836b-85805c7d1b52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:45:32.141358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb3e8c3-eed7-468f-836b-85805c7d1b52 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:45:32.151225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 763/1530] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:45:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:45:34.159632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fb2c2f-b422-4ce8-8079-e8d4a17abc74 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:34.160239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fb2c2f-b422-4ce8-8079-e8d4a17abc74 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:34.160239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fb2c2f-b422-4ce8-8079-e8d4a17abc74 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:34.161580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 768/1531] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:45:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:45:34.167463 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c7f302-06bb-420d-98df-2f2775877052 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:34.167924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c7f302-06bb-420d-98df-2f2775877052 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:34.168043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c7f302-06bb-420d-98df-2f2775877052 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:34.187761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-19c7f302-06bb-420d-98df-2f2775877052 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:45:34.188992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-19c7f302-06bb-420d-98df-2f2775877052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:34.196659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 764/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:45:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:45:34.232545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fb2c2f-b422-4ce8-8079-e8d4a17abc74 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:45:34.243426 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fb2c2f-b422-4ce8-8079-e8d4a17abc74 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:45:34.247578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204e1a14-5955-4387-b941-79e8830f1ac4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:45:34.248180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204e1a14-5955-4387-b941-79e8830f1ac4 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:34.248463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204e1a14-5955-4387-b941-79e8830f1ac4 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:34.248777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204e1a14-5955-4387-b941-79e8830f1ac4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:34.257415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204e1a14-5955-4387-b941-79e8830f1ac4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:45:34.258837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204e1a14-5955-4387-b941-79e8830f1ac4 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:45:34.269049 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 769/1533] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:45:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:45:37.826257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53441e21-1a35-4199-8aae-7187317af440 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:37.826783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53441e21-1a35-4199-8aae-7187317af440 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:37.826783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53441e21-1a35-4199-8aae-7187317af440 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:37.828101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 765/1534] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:45:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:45:37.833257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0ef4f8-627a-4720-b4ef-5a5beda12761 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:37.833653 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0ef4f8-627a-4720-b4ef-5a5beda12761 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:37.833812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0ef4f8-627a-4720-b4ef-5a5beda12761 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:37.854110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-da0ef4f8-627a-4720-b4ef-5a5beda12761 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:45:37.855298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-da0ef4f8-627a-4720-b4ef-5a5beda12761 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:37.863020 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 770/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:45:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:45:38.168255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53441e21-1a35-4199-8aae-7187317af440 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:45:38.169938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-53441e21-1a35-4199-8aae-7187317af440 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:38.234021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:45:38.247896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53441e21-1a35-4199-8aae-7187317af440 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:45:38.251458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b32b30-edca-4017-901d-8464096340b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:45:38.252077 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b32b30-edca-4017-901d-8464096340b5 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:38.252359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b32b30-edca-4017-901d-8464096340b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:38.252659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b32b30-edca-4017-901d-8464096340b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:38.262319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b32b30-edca-4017-901d-8464096340b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:45:38.263762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b32b30-edca-4017-901d-8464096340b5 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:45:38.273219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 766/1536] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:45:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:45:40.315652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d901c1-25c3-4f27-84a0-d55383316bd1 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:40.316217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d901c1-25c3-4f27-84a0-d55383316bd1 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:40.316217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d901c1-25c3-4f27-84a0-d55383316bd1 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:40.317619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 771/1537] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jul 29 04:45:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:45:40.322809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85049a-92d1-496e-8a8e-a0c69d23d094 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:40.323251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85049a-92d1-496e-8a8e-a0c69d23d094 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:40.323416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85049a-92d1-496e-8a8e-a0c69d23d094 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:40.343181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-eb85049a-92d1-496e-8a8e-a0c69d23d094 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:45:40.344405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-eb85049a-92d1-496e-8a8e-a0c69d23d094 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:40.352306 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 767/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jul 29 04:45:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:45:40.667438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d901c1-25c3-4f27-84a0-d55383316bd1 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:45:40.669160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-65d901c1-25c3-4f27-84a0-d55383316bd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:45:40.677064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=84 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:45:40.684095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d901c1-25c3-4f27-84a0-d55383316bd1 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:45:40.687618 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d4a891-a07f-4a5e-a71e-97bfe97dcc5b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:45:40.688237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d4a891-a07f-4a5e-a71e-97bfe97dcc5b service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:45:40.688523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d4a891-a07f-4a5e-a71e-97bfe97dcc5b service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:45:40.688833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d4a891-a07f-4a5e-a71e-97bfe97dcc5b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:45:40.698284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d4a891-a07f-4a5e-a71e-97bfe97dcc5b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:45:40.699719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d4a891-a07f-4a5e-a71e-97bfe97dcc5b service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:45:40.709050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 772/1539] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:45:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:45:50.338080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=11 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:45:50.704942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:12.818409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97136ccc-9fd0-43f3-ac5e-43363c469cfb None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:12.819807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97136ccc-9fd0-43f3-ac5e-43363c469cfb None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:12.819807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97136ccc-9fd0-43f3-ac5e-43363c469cfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:12.853687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-97136ccc-9fd0-43f3-ac5e-43363c469cfb None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:12.855227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-97136ccc-9fd0-43f3-ac5e-43363c469cfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:12.866938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 768/1540] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:12.876370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c12005-469d-49a5-88c7-3b1b6319c7f7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:12.876955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c12005-469d-49a5-88c7-3b1b6319c7f7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:12.877163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c12005-469d-49a5-88c7-3b1b6319c7f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:12.909879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-13c12005-469d-49a5-88c7-3b1b6319c7f7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:12.911171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-13c12005-469d-49a5-88c7-3b1b6319c7f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:12.927211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 773/1541] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:12.933275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97136ccc-9fd0-43f3-ac5e-43363c469cfb None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:12.939795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7f4c2b-7190-4497-9837-52b34a195f05 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:12.940215 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7f4c2b-7190-4497-9837-52b34a195f05 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:12.940498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7f4c2b-7190-4497-9837-52b34a195f05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:12.948235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97136ccc-9fd0-43f3-ac5e-43363c469cfb None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:12.955807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f49b6ff-6184-4623-b928-7cc3300af59f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:12.956892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f49b6ff-6184-4623-b928-7cc3300af59f admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:12.957412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f49b6ff-6184-4623-b928-7cc3300af59f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:12.957920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f49b6ff-6184-4623-b928-7cc3300af59f admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:12.958852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f49b6ff-6184-4623-b928-7cc3300af59f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:12.959432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f49b6ff-6184-4623-b928-7cc3300af59f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:12.961307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f49b6ff-6184-4623-b928-7cc3300af59f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:12.977539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 769/1542] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:12.977880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-ee7f4c2b-7190-4497-9837-52b34a195f05 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:12.979297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ee7f4c2b-7190-4497-9837-52b34a195f05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:12.979439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1be8a31-3692-4967-9c48-6c3483f5e809 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:12.979790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1be8a31-3692-4967-9c48-6c3483f5e809 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:12.980170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1be8a31-3692-4967-9c48-6c3483f5e809 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:12.988644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 774/1543] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Jul 29 04:47:12.998581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61460142-b4fa-4e96-a890-6e5016bbb121 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:12.999122 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61460142-b4fa-4e96-a890-6e5016bbb121 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:12.999561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61460142-b4fa-4e96-a890-6e5016bbb121 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.013418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d1be8a31-3692-4967-9c48-6c3483f5e809 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.014784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d1be8a31-3692-4967-9c48-6c3483f5e809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.026779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-61460142-b4fa-4e96-a890-6e5016bbb121 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.028485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-61460142-b4fa-4e96-a890-6e5016bbb121 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.030132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 770/1544] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.031274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1be8a31-3692-4967-9c48-6c3483f5e809 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.046321 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 775/1545] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:13.047501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1be8a31-3692-4967-9c48-6c3483f5e809 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.048304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c0c944-00a7-44d3-b66b-27df25a4f3fd None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.048841 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c0c944-00a7-44d3-b66b-27df25a4f3fd None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.049272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c0c944-00a7-44d3-b66b-27df25a4f3fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.062211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c2d8f77-2154-4cb1-aef6-a81ed75bb1dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.063180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2d8f77-2154-4cb1-aef6-a81ed75bb1dd admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.063663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2d8f77-2154-4cb1-aef6-a81ed75bb1dd admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.064135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2d8f77-2154-4cb1-aef6-a81ed75bb1dd admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.065020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2d8f77-2154-4cb1-aef6-a81ed75bb1dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.066245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WARNING py.warnings [None req-7c2d8f77-2154-4cb1-aef6-a81ed75bb1dd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:47:13.066245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: warnings.warn(msg) Jul 29 04:47:13.066245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: Jul 29 04:47:13.067793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c2d8f77-2154-4cb1-aef6-a81ed75bb1dd admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.073957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-26c0c944-00a7-44d3-b66b-27df25a4f3fd None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.075392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-26c0c944-00a7-44d3-b66b-27df25a4f3fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.089640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 776/1546] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.091161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917ba0cf-a988-48ca-a91b-b8597a967dd2 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.093829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917ba0cf-a988-48ca-a91b-b8597a967dd2 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.094291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917ba0cf-a988-48ca-a91b-b8597a967dd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.126741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-917ba0cf-a988-48ca-a91b-b8597a967dd2 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.128160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-917ba0cf-a988-48ca-a91b-b8597a967dd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.141520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 771/1548] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/projects => generated 393 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.143329 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940cd890-8fd5-41bb-b377-5e770f9d7726 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.143829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940cd890-8fd5-41bb-b377-5e770f9d7726 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.144314 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940cd890-8fd5-41bb-b377-5e770f9d7726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.144812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 777/1548] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.146042 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917ba0cf-a988-48ca-a91b-b8597a967dd2 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.160951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917ba0cf-a988-48ca-a91b-b8597a967dd2 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.166112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-940cd890-8fd5-41bb-b377-5e770f9d7726 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.167344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-940cd890-8fd5-41bb-b377-5e770f9d7726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.222180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048e05c8-e09b-405e-87f0-fb1d3292cbc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.222810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048e05c8-e09b-405e-87f0-fb1d3292cbc3 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.223101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048e05c8-e09b-405e-87f0-fb1d3292cbc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.223385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048e05c8-e09b-405e-87f0-fb1d3292cbc3 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.224089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048e05c8-e09b-405e-87f0-fb1d3292cbc3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:13.224399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048e05c8-e09b-405e-87f0-fb1d3292cbc3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.225587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: WARNING py.warnings [None req-048e05c8-e09b-405e-87f0-fb1d3292cbc3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:47:13.225587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: warnings.warn(msg) Jul 29 04:47:13.225587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: Jul 29 04:47:13.226481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048e05c8-e09b-405e-87f0-fb1d3292cbc3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.228273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 772/1549] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.229240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940cd890-8fd5-41bb-b377-5e770f9d7726 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.240049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940cd890-8fd5-41bb-b377-5e770f9d7726 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.242719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 778/1550] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:13.244095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c0b877-3300-43e2-af44-4c7057839def None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.244394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c0b877-3300-43e2-af44-4c7057839def None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.244706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c0b877-3300-43e2-af44-4c7057839def None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.244887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a88da72-4289-496a-a0d1-89a6e7f4a280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.245499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a88da72-4289-496a-a0d1-89a6e7f4a280 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.245786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a88da72-4289-496a-a0d1-89a6e7f4a280 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.246055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a88da72-4289-496a-a0d1-89a6e7f4a280 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.246757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a88da72-4289-496a-a0d1-89a6e7f4a280 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:13.247060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a88da72-4289-496a-a0d1-89a6e7f4a280 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.248760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a88da72-4289-496a-a0d1-89a6e7f4a280 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.261269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 773/1551] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:13.262241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e56584-41e9-4add-bacb-344bd747dda1 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.262533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e56584-41e9-4add-bacb-344bd747dda1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.262813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e56584-41e9-4add-bacb-344bd747dda1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.272484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-01c0b877-3300-43e2-af44-4c7057839def None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.273745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-01c0b877-3300-43e2-af44-4c7057839def None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.281855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 779/1552] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.282283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-b0e56584-41e9-4add-bacb-344bd747dda1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.282688 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c0b877-3300-43e2-af44-4c7057839def None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.283464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b0e56584-41e9-4add-bacb-344bd747dda1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.291818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 774/1553] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.293002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c0b877-3300-43e2-af44-4c7057839def None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.293110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8428c4d-6a47-486c-a1f2-fc1469acdb84 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.293343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8428c4d-6a47-486c-a1f2-fc1469acdb84 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.293619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8428c4d-6a47-486c-a1f2-fc1469acdb84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.298293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d291d7-4775-47a7-a305-0c399c8aa6b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.299002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d291d7-4775-47a7-a305-0c399c8aa6b2 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.299843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d291d7-4775-47a7-a305-0c399c8aa6b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.300029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d291d7-4775-47a7-a305-0c399c8aa6b2 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.300761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d291d7-4775-47a7-a305-0c399c8aa6b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.301842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: WARNING py.warnings [None req-24d291d7-4775-47a7-a305-0c399c8aa6b2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:47:13.301842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: warnings.warn(msg) Jul 29 04:47:13.301842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: Jul 29 04:47:13.303065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d291d7-4775-47a7-a305-0c399c8aa6b2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.314514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a8428c4d-6a47-486c-a1f2-fc1469acdb84 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.315875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a8428c4d-6a47-486c-a1f2-fc1469acdb84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.325080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 775/1554] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.325221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 780/1555] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/users => generated 297 bytes in 43 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.326258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8ef940-709d-40cd-8f4c-c5f5e0faf204 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.326551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8ef940-709d-40cd-8f4c-c5f5e0faf204 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.326833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8ef940-709d-40cd-8f4c-c5f5e0faf204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.329400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5d2199-de9d-4469-b81e-e549124c8576 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.329692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5d2199-de9d-4469-b81e-e549124c8576 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.329976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5d2199-de9d-4469-b81e-e549124c8576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.348152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-cd8ef940-709d-40cd-8f4c-c5f5e0faf204 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.349045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-dd5d2199-de9d-4469-b81e-e549124c8576 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.349413 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-cd8ef940-709d-40cd-8f4c-c5f5e0faf204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.350270 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-dd5d2199-de9d-4469-b81e-e549124c8576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.357484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 781/1556] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.358255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 776/1557] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.358827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5d2199-de9d-4469-b81e-e549124c8576 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.360910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8ef940-709d-40cd-8f4c-c5f5e0faf204 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.368461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5d2199-de9d-4469-b81e-e549124c8576 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.372312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8ef940-709d-40cd-8f4c-c5f5e0faf204 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.373395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd277d72-7d83-4572-90b9-0dbb08f5d385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.374533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd277d72-7d83-4572-90b9-0dbb08f5d385 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.374533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd277d72-7d83-4572-90b9-0dbb08f5d385 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.374533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd277d72-7d83-4572-90b9-0dbb08f5d385 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.375204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd277d72-7d83-4572-90b9-0dbb08f5d385 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.377147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd277d72-7d83-4572-90b9-0dbb08f5d385 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.377884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3480a805-3f79-4624-9d54-a8de9b165773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.378505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3480a805-3f79-4624-9d54-a8de9b165773 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.378795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3480a805-3f79-4624-9d54-a8de9b165773 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.379078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3480a805-3f79-4624-9d54-a8de9b165773 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.379783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3480a805-3f79-4624-9d54-a8de9b165773 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.382285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3480a805-3f79-4624-9d54-a8de9b165773 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.406659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 777/1558] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.407989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8645b6-a285-4a65-96c8-17766c8bdb51 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.408417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8645b6-a285-4a65-96c8-17766c8bdb51 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.408714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8645b6-a285-4a65-96c8-17766c8bdb51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.410642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 782/1559] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/projects => generated 379 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.412182 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cb545e-a034-43dd-b05b-7909426b2e55 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.412573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cb545e-a034-43dd-b05b-7909426b2e55 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.412914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cb545e-a034-43dd-b05b-7909426b2e55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.428729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-ca8645b6-a285-4a65-96c8-17766c8bdb51 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.430090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ca8645b6-a285-4a65-96c8-17766c8bdb51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.432907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f5cb545e-a034-43dd-b05b-7909426b2e55 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.434090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f5cb545e-a034-43dd-b05b-7909426b2e55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.474109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:13.491243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 778/1560] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.492297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8645b6-a285-4a65-96c8-17766c8bdb51 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.497378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 783/1561] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.498406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cb545e-a034-43dd-b05b-7909426b2e55 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.502349 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8645b6-a285-4a65-96c8-17766c8bdb51 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.507186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a3d97-9385-4aa7-a87f-ff8886ad8db7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.507812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a3d97-9385-4aa7-a87f-ff8886ad8db7 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.507968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cb545e-a034-43dd-b05b-7909426b2e55 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.508099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a3d97-9385-4aa7-a87f-ff8886ad8db7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.508382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a3d97-9385-4aa7-a87f-ff8886ad8db7 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.509129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533a3d97-9385-4aa7-a87f-ff8886ad8db7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.510733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533a3d97-9385-4aa7-a87f-ff8886ad8db7 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.512759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b479f7ae-c096-4757-8d3a-42fb85153f69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.513375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b479f7ae-c096-4757-8d3a-42fb85153f69 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.513662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b479f7ae-c096-4757-8d3a-42fb85153f69 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.513928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b479f7ae-c096-4757-8d3a-42fb85153f69 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.514628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b479f7ae-c096-4757-8d3a-42fb85153f69 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:13.514932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b479f7ae-c096-4757-8d3a-42fb85153f69 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.516532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b479f7ae-c096-4757-8d3a-42fb85153f69 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.519038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:13.523533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 779/1562] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:13.524628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a3d97-9385-4aa7-a87f-ff8886ad8db7 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.532676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 784/1563] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:13.534125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971bff38-55c5-40d5-9f25-0fcad10b35f4 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.534414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971bff38-55c5-40d5-9f25-0fcad10b35f4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.534695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971bff38-55c5-40d5-9f25-0fcad10b35f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.536854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a3d97-9385-4aa7-a87f-ff8886ad8db7 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.544520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a069f0fb-2407-40e4-9d3d-6d892d617b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.545265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069f0fb-2407-40e4-9d3d-6d892d617b91 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.545551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069f0fb-2407-40e4-9d3d-6d892d617b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.545834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a069f0fb-2407-40e4-9d3d-6d892d617b91 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.546785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a069f0fb-2407-40e4-9d3d-6d892d617b91 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:13.547095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a069f0fb-2407-40e4-9d3d-6d892d617b91 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.548749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a069f0fb-2407-40e4-9d3d-6d892d617b91 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.555252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-971bff38-55c5-40d5-9f25-0fcad10b35f4 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:13.556448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-971bff38-55c5-40d5-9f25-0fcad10b35f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:13.568659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 780/1564] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:13.568659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a069f0fb-2407-40e4-9d3d-6d892d617b91 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.571543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 785/1565] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:13.571810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a069f0fb-2407-40e4-9d3d-6d892d617b91 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.572668 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971bff38-55c5-40d5-9f25-0fcad10b35f4 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.578853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce495ad-5007-46bb-8af4-2001e90a7f53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.579465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce495ad-5007-46bb-8af4-2001e90a7f53 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.579745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce495ad-5007-46bb-8af4-2001e90a7f53 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.580020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce495ad-5007-46bb-8af4-2001e90a7f53 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.580724 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce495ad-5007-46bb-8af4-2001e90a7f53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.582487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971bff38-55c5-40d5-9f25-0fcad10b35f4 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.582645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce495ad-5007-46bb-8af4-2001e90a7f53 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.587714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f22173-0e10-441b-a820-5409e2fcd620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.588346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f22173-0e10-441b-a820-5409e2fcd620 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.588663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f22173-0e10-441b-a820-5409e2fcd620 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.588949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f22173-0e10-441b-a820-5409e2fcd620 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.589612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f22173-0e10-441b-a820-5409e2fcd620 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.591516 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f22173-0e10-441b-a820-5409e2fcd620 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.606415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 781/1566] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/users => generated 287 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.607246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce495ad-5007-46bb-8af4-2001e90a7f53 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.614461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 786/1567] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/users => generated 290 bytes in 42 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.615505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f22173-0e10-441b-a820-5409e2fcd620 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.619958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce495ad-5007-46bb-8af4-2001e90a7f53 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.624648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daee2195-ad6c-44db-9af7-38c144445c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.624932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f22173-0e10-441b-a820-5409e2fcd620 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.625397 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daee2195-ad6c-44db-9af7-38c144445c10 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.625760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daee2195-ad6c-44db-9af7-38c144445c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.626067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daee2195-ad6c-44db-9af7-38c144445c10 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.626759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daee2195-ad6c-44db-9af7-38c144445c10 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:13.627064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daee2195-ad6c-44db-9af7-38c144445c10 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.628684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daee2195-ad6c-44db-9af7-38c144445c10 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.629657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d3669b-b82c-4b7a-9845-8669eb1e4dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.630251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d3669b-b82c-4b7a-9845-8669eb1e4dde admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.630532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d3669b-b82c-4b7a-9845-8669eb1e4dde admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.630812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d3669b-b82c-4b7a-9845-8669eb1e4dde admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.631527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d3669b-b82c-4b7a-9845-8669eb1e4dde admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:13.631830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d3669b-b82c-4b7a-9845-8669eb1e4dde admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.633448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d3669b-b82c-4b7a-9845-8669eb1e4dde admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.641066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 782/1568] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:13.641859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daee2195-ad6c-44db-9af7-38c144445c10 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.651996 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 787/1569] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:13.652925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d3669b-b82c-4b7a-9845-8669eb1e4dde admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.661097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daee2195-ad6c-44db-9af7-38c144445c10 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.664079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d3669b-b82c-4b7a-9845-8669eb1e4dde admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.666654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6d9e02-0220-4e74-b930-1bd547571aca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.667316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6d9e02-0220-4e74-b930-1bd547571aca admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.667639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6d9e02-0220-4e74-b930-1bd547571aca admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.667904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6d9e02-0220-4e74-b930-1bd547571aca admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.668637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6d9e02-0220-4e74-b930-1bd547571aca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.668848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16b629e-d0f4-4726-b722-65ed148115be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.669457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16b629e-d0f4-4726-b722-65ed148115be admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.669737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16b629e-d0f4-4726-b722-65ed148115be admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.670010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16b629e-d0f4-4726-b722-65ed148115be admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.670289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6d9e02-0220-4e74-b930-1bd547571aca admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.670668 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16b629e-d0f4-4726-b722-65ed148115be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.672842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16b629e-d0f4-4726-b722-65ed148115be admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.681997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 783/1570] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:13.682802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6d9e02-0220-4e74-b930-1bd547571aca admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.693044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6d9e02-0220-4e74-b930-1bd547571aca admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.698567 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6250ffc6-e44a-46d9-a833-78022b723ca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.699182 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6250ffc6-e44a-46d9-a833-78022b723ca6 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.699496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6250ffc6-e44a-46d9-a833-78022b723ca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.699782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6250ffc6-e44a-46d9-a833-78022b723ca6 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.700444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6250ffc6-e44a-46d9-a833-78022b723ca6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.702388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6250ffc6-e44a-46d9-a833-78022b723ca6 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.702499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 788/1571] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.703217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16b629e-d0f4-4726-b722-65ed148115be admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.738200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 784/1572] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/projects => generated 385 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.738200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6250ffc6-e44a-46d9-a833-78022b723ca6 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.770415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16b629e-d0f4-4726-b722-65ed148115be admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.799791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6250ffc6-e44a-46d9-a833-78022b723ca6 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.804542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151360b5-4c9e-46d2-8a24-f0f81a753df3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.805201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151360b5-4c9e-46d2-8a24-f0f81a753df3 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.805490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151360b5-4c9e-46d2-8a24-f0f81a753df3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.805771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151360b5-4c9e-46d2-8a24-f0f81a753df3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.806485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151360b5-4c9e-46d2-8a24-f0f81a753df3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.808106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151360b5-4c9e-46d2-8a24-f0f81a753df3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.818477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 785/1573] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:13.819897 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151360b5-4c9e-46d2-8a24-f0f81a753df3 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.829549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151360b5-4c9e-46d2-8a24-f0f81a753df3 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.830101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2847bf9-f54b-4857-afea-c5133d61802e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.830761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2847bf9-f54b-4857-afea-c5133d61802e admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.831051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2847bf9-f54b-4857-afea-c5133d61802e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.831362 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2847bf9-f54b-4857-afea-c5133d61802e admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.832074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2847bf9-f54b-4857-afea-c5133d61802e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:13.832388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2847bf9-f54b-4857-afea-c5133d61802e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.834161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2847bf9-f54b-4857-afea-c5133d61802e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.835024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c84c8cf-5d80-4811-a260-3f05bbf0e8e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.835694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c84c8cf-5d80-4811-a260-3f05bbf0e8e0 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.836054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c84c8cf-5d80-4811-a260-3f05bbf0e8e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.836406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c84c8cf-5d80-4811-a260-3f05bbf0e8e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.837199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c84c8cf-5d80-4811-a260-3f05bbf0e8e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.838784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c84c8cf-5d80-4811-a260-3f05bbf0e8e0 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.846319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 789/1574] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:13.853584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2847bf9-f54b-4857-afea-c5133d61802e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.853704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 786/1575] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:13.853704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c84c8cf-5d80-4811-a260-3f05bbf0e8e0 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.857833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2847bf9-f54b-4857-afea-c5133d61802e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.861094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c84c8cf-5d80-4811-a260-3f05bbf0e8e0 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.862597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c94080-27cd-459d-8c86-a8154278d0b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.863223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c94080-27cd-459d-8c86-a8154278d0b6 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.863508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c94080-27cd-459d-8c86-a8154278d0b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.863785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c94080-27cd-459d-8c86-a8154278d0b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.864497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c94080-27cd-459d-8c86-a8154278d0b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.865991 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b42096-2cf2-429c-a5a8-5154fc812fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.866443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c94080-27cd-459d-8c86-a8154278d0b6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.866589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b42096-2cf2-429c-a5a8-5154fc812fbc admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.866877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b42096-2cf2-429c-a5a8-5154fc812fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.867156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b42096-2cf2-429c-a5a8-5154fc812fbc admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.867820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b42096-2cf2-429c-a5a8-5154fc812fbc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.869817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b42096-2cf2-429c-a5a8-5154fc812fbc admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.900193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 790/1576] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/projects => generated 383 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.900831 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 787/1577] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.900947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c94080-27cd-459d-8c86-a8154278d0b6 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.901055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b42096-2cf2-429c-a5a8-5154fc812fbc admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:13.918485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:13.927996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:13.965245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b42096-2cf2-429c-a5a8-5154fc812fbc admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.967303 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c94080-27cd-459d-8c86-a8154278d0b6 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:13.969949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3468fc05-aabc-46e7-9953-456daf317200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.970571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3468fc05-aabc-46e7-9953-456daf317200 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.970849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3468fc05-aabc-46e7-9953-456daf317200 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.971130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3468fc05-aabc-46e7-9953-456daf317200 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.971787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3468fc05-aabc-46e7-9953-456daf317200 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.972128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6845c1b-3fce-4f6d-ab79-441914261ceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:13.972828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6845c1b-3fce-4f6d-ab79-441914261ceb admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:13.973116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6845c1b-3fce-4f6d-ab79-441914261ceb admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:13.973395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6845c1b-3fce-4f6d-ab79-441914261ceb admin admin] PATH_INFO: `/v3/projects/0f6c1ad3422c40d4af4ef5329ecd9d86/users/5f9d45265c6f47609f9f8abcf3526ff0/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:13.973740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3468fc05-aabc-46e7-9953-456daf317200 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.989431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6845c1b-3fce-4f6d-ab79-441914261ceb admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f6c1ad3422c40d4af4ef5329ecd9d86, user_id=5f9d45265c6f47609f9f8abcf3526ff0, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:13.992414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6845c1b-3fce-4f6d-ab79-441914261ceb admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:13.996475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 788/1578] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/users => generated 287 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:13.997102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3468fc05-aabc-46e7-9953-456daf317200 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.007295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3468fc05-aabc-46e7-9953-456daf317200 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.007571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 791/1579] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:13 2022] PUT /identity/v3/projects/0f6c1ad3422c40d4af4ef5329ecd9d86/users/5f9d45265c6f47609f9f8abcf3526ff0/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:14.008479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6845c1b-3fce-4f6d-ab79-441914261ceb admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.066615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b49b85-a520-433e-93d5-541bcd36e7e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.067245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b49b85-a520-433e-93d5-541bcd36e7e8 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.067542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b49b85-a520-433e-93d5-541bcd36e7e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.067821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b49b85-a520-433e-93d5-541bcd36e7e8 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.068490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b49b85-a520-433e-93d5-541bcd36e7e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.070599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b49b85-a520-433e-93d5-541bcd36e7e8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.072811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6845c1b-3fce-4f6d-ab79-441914261ceb admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.077431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd7d74-a9b4-43de-98a7-d00b4ac8a7d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.078049 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd7d74-a9b4-43de-98a7-d00b4ac8a7d3 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.078339 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd7d74-a9b4-43de-98a7-d00b4ac8a7d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.078617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fd7d74-a9b4-43de-98a7-d00b4ac8a7d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.079320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fd7d74-a9b4-43de-98a7-d00b4ac8a7d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.080970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fd7d74-a9b4-43de-98a7-d00b4ac8a7d3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.092778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 792/1580] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.092946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 789/1581] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:13 2022] POST /identity/v3/users => generated 293 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:14.093726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b49b85-a520-433e-93d5-541bcd36e7e8 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.094050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd7d74-a9b4-43de-98a7-d00b4ac8a7d3 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.104147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fd7d74-a9b4-43de-98a7-d00b4ac8a7d3 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.106622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b49b85-a520-433e-93d5-541bcd36e7e8 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.108966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871e28af-8cb1-4fd6-ae76-b0477cde7a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.109629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e28af-8cb1-4fd6-ae76-b0477cde7a89 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.109917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e28af-8cb1-4fd6-ae76-b0477cde7a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.110195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871e28af-8cb1-4fd6-ae76-b0477cde7a89 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.110883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871e28af-8cb1-4fd6-ae76-b0477cde7a89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.111596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458863f0-9061-43f0-9429-5bf099eda2c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.112211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458863f0-9061-43f0-9429-5bf099eda2c1 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.112502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458863f0-9061-43f0-9429-5bf099eda2c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.112825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871e28af-8cb1-4fd6-ae76-b0477cde7a89 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.112941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458863f0-9061-43f0-9429-5bf099eda2c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.113527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458863f0-9061-43f0-9429-5bf099eda2c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.115112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458863f0-9061-43f0-9429-5bf099eda2c1 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.126002 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 790/1582] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.126873 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458863f0-9061-43f0-9429-5bf099eda2c1 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.139542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458863f0-9061-43f0-9429-5bf099eda2c1 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.141476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 793/1583] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:14 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:14.142478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871e28af-8cb1-4fd6-ae76-b0477cde7a89 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.199887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6977e4-4e9c-4ac0-bfed-d0ab667c6ed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.200489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6977e4-4e9c-4ac0-bfed-d0ab667c6ed7 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.200758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6977e4-4e9c-4ac0-bfed-d0ab667c6ed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.201050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6977e4-4e9c-4ac0-bfed-d0ab667c6ed7 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.201719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6977e4-4e9c-4ac0-bfed-d0ab667c6ed7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.203749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6977e4-4e9c-4ac0-bfed-d0ab667c6ed7 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.208194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871e28af-8cb1-4fd6-ae76-b0477cde7a89 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.213541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677ce0d7-2594-4f1a-b14e-14c46cb3ac7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.214279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677ce0d7-2594-4f1a-b14e-14c46cb3ac7c admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.214575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677ce0d7-2594-4f1a-b14e-14c46cb3ac7c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.214913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677ce0d7-2594-4f1a-b14e-14c46cb3ac7c admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.215533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677ce0d7-2594-4f1a-b14e-14c46cb3ac7c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.217618 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677ce0d7-2594-4f1a-b14e-14c46cb3ac7c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.235657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 791/1584] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:14 2022] POST /identity/v3/users => generated 287 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:14.236494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6977e4-4e9c-4ac0-bfed-d0ab667c6ed7 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.247599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6977e4-4e9c-4ac0-bfed-d0ab667c6ed7 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.247767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 794/1585] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:14 2022] POST /identity/v3/users => generated 292 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:14.249107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991199e-2da1-4464-b366-e801cf330d0e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.249447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991199e-2da1-4464-b366-e801cf330d0e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.249843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e991199e-2da1-4464-b366-e801cf330d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.255177 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f729c7b0-2304-4fd1-ae4b-26361d30feb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.255795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f729c7b0-2304-4fd1-ae4b-26361d30feb7 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.256120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f729c7b0-2304-4fd1-ae4b-26361d30feb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.257032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f729c7b0-2304-4fd1-ae4b-26361d30feb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.257897 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f729c7b0-2304-4fd1-ae4b-26361d30feb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.258891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:14.259517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f729c7b0-2304-4fd1-ae4b-26361d30feb7 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.270197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 792/1586] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.271026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f729c7b0-2304-4fd1-ae4b-26361d30feb7 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.276199 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e991199e-2da1-4464-b366-e801cf330d0e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:14.277485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e991199e-2da1-4464-b366-e801cf330d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:14.280898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f729c7b0-2304-4fd1-ae4b-26361d30feb7 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.285371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 795/1587] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:14.285585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eb48df-7b80-4574-b33d-2bfe213d1a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.286192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e991199e-2da1-4464-b366-e801cf330d0e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.286496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eb48df-7b80-4574-b33d-2bfe213d1a59 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.286753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eb48df-7b80-4574-b33d-2bfe213d1a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.287042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66eb48df-7b80-4574-b33d-2bfe213d1a59 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.287905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66eb48df-7b80-4574-b33d-2bfe213d1a59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.290227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66eb48df-7b80-4574-b33d-2bfe213d1a59 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.297731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:INSERT=9 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:14.303644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e991199e-2da1-4464-b366-e801cf330d0e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.308636 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 793/1588] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.308765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b23874-9be3-4a6c-aa83-809bc2eeab6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.309274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b23874-9be3-4a6c-aa83-809bc2eeab6a admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.309605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b23874-9be3-4a6c-aa83-809bc2eeab6a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.309875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b23874-9be3-4a6c-aa83-809bc2eeab6a admin admin] PATH_INFO: `/v3/projects/2cd7d10d7e2a42ca9afa3ea77a420848/users/00463dc0723f414fbd55018d79febc66/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.310403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eb48df-7b80-4574-b33d-2bfe213d1a59 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.324039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66eb48df-7b80-4574-b33d-2bfe213d1a59 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.328909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a5607c-b1ed-40a3-b023-efb366c7df19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.329066 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b23874-9be3-4a6c-aa83-809bc2eeab6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=2cd7d10d7e2a42ca9afa3ea77a420848, user_id=00463dc0723f414fbd55018d79febc66, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.329578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a5607c-b1ed-40a3-b023-efb366c7df19 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.329860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a5607c-b1ed-40a3-b023-efb366c7df19 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.330134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a5607c-b1ed-40a3-b023-efb366c7df19 admin admin] PATH_INFO: `/v3/projects/77688c4faed945849f5649c08cc580c2/users/34242a2360f14710af2e8dbe6b704ca1/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.331844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b23874-9be3-4a6c-aa83-809bc2eeab6a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.345641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a5607c-b1ed-40a3-b023-efb366c7df19 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77688c4faed945849f5649c08cc580c2, user_id=34242a2360f14710af2e8dbe6b704ca1, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.345844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 796/1589] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:14 2022] PUT /identity/v3/projects/2cd7d10d7e2a42ca9afa3ea77a420848/users/00463dc0723f414fbd55018d79febc66/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:14.346844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b23874-9be3-4a6c-aa83-809bc2eeab6a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.357352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WARNING py.warnings [None req-98a5607c-b1ed-40a3-b023-efb366c7df19 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:47:14.357352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: warnings.warn(msg) Jul 29 04:47:14.357352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: Jul 29 04:47:14.359312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a5607c-b1ed-40a3-b023-efb366c7df19 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.374029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 794/1590] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:14 2022] PUT /identity/v3/projects/77688c4faed945849f5649c08cc580c2/users/34242a2360f14710af2e8dbe6b704ca1/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:14.377509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a5607c-b1ed-40a3-b023-efb366c7df19 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.411606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b23874-9be3-4a6c-aa83-809bc2eeab6a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.442925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a5607c-b1ed-40a3-b023-efb366c7df19 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.448233 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fdff87-ed7c-4617-bd94-5707be46a8b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.448887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fdff87-ed7c-4617-bd94-5707be46a8b8 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.449169 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fdff87-ed7c-4617-bd94-5707be46a8b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.449443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fdff87-ed7c-4617-bd94-5707be46a8b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.450197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fdff87-ed7c-4617-bd94-5707be46a8b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.451730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fdff87-ed7c-4617-bd94-5707be46a8b8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.461785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 795/1591] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.462876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fdff87-ed7c-4617-bd94-5707be46a8b8 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.468433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9224c41-08aa-4aaa-8e3b-fdbef70a9b8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.469088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9224c41-08aa-4aaa-8e3b-fdbef70a9b8c admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.469375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9224c41-08aa-4aaa-8e3b-fdbef70a9b8c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.469651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9224c41-08aa-4aaa-8e3b-fdbef70a9b8c admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.470312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9224c41-08aa-4aaa-8e3b-fdbef70a9b8c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.472237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9224c41-08aa-4aaa-8e3b-fdbef70a9b8c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.472779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fdff87-ed7c-4617-bd94-5707be46a8b8 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.477511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c858cf-8861-4f68-9e62-d05b4009a966 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.478099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c858cf-8861-4f68-9e62-d05b4009a966 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.478387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c858cf-8861-4f68-9e62-d05b4009a966 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.478668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c858cf-8861-4f68-9e62-d05b4009a966 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.479415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c858cf-8861-4f68-9e62-d05b4009a966 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.481021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c858cf-8861-4f68-9e62-d05b4009a966 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.495627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 796/1592] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.495627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c858cf-8861-4f68-9e62-d05b4009a966 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.495917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 797/1593] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:14 2022] POST /identity/v3/users => generated 292 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:14.495917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9224c41-08aa-4aaa-8e3b-fdbef70a9b8c admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.502826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c858cf-8861-4f68-9e62-d05b4009a966 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.508222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936700a7-33ff-4c74-851a-8e3bb9d503b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.508853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936700a7-33ff-4c74-851a-8e3bb9d503b4 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.509141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936700a7-33ff-4c74-851a-8e3bb9d503b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.509419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936700a7-33ff-4c74-851a-8e3bb9d503b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.510161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936700a7-33ff-4c74-851a-8e3bb9d503b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.511751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936700a7-33ff-4c74-851a-8e3bb9d503b4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.522656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 797/1594] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.522656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936700a7-33ff-4c74-851a-8e3bb9d503b4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.532955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936700a7-33ff-4c74-851a-8e3bb9d503b4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.538082 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106e5630-c6ce-4cae-996d-0d5be3418449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.538679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106e5630-c6ce-4cae-996d-0d5be3418449 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.538959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106e5630-c6ce-4cae-996d-0d5be3418449 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.539231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106e5630-c6ce-4cae-996d-0d5be3418449 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.539942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106e5630-c6ce-4cae-996d-0d5be3418449 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.541644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106e5630-c6ce-4cae-996d-0d5be3418449 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.553830 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 798/1595] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.555017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e483802-b9cf-4377-b1c2-978f187b7e7b None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.555259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9224c41-08aa-4aaa-8e3b-fdbef70a9b8c admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.555358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e483802-b9cf-4377-b1c2-978f187b7e7b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.555572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e483802-b9cf-4377-b1c2-978f187b7e7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.559101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa90804c-c49b-4a93-95cc-910214532005 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.559837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90804c-c49b-4a93-95cc-910214532005 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.560056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90804c-c49b-4a93-95cc-910214532005 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.560334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa90804c-c49b-4a93-95cc-910214532005 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.570710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa90804c-c49b-4a93-95cc-910214532005 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.572177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa90804c-c49b-4a93-95cc-910214532005 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.581998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 798/1596] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:14.583041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-4e483802-b9cf-4377-b1c2-978f187b7e7b None None] MFA Rules not processed for user `00463dc0723f414fbd55018d79febc66`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:14.583410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409cb553-abe5-491d-94c9-6780806d0559 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.583703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409cb553-abe5-491d-94c9-6780806d0559 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.583981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409cb553-abe5-491d-94c9-6780806d0559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.584822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-4e483802-b9cf-4377-b1c2-978f187b7e7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:14.604705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-409cb553-abe5-491d-94c9-6780806d0559 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:14.605928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-409cb553-abe5-491d-94c9-6780806d0559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:14.614323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 799/1597] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:14.615060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409cb553-abe5-491d-94c9-6780806d0559 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.622561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:14.626414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409cb553-abe5-491d-94c9-6780806d0559 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.634072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b690f57-5989-4190-a126-4a110320bf43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.634788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b690f57-5989-4190-a126-4a110320bf43 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.635146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b690f57-5989-4190-a126-4a110320bf43 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.635576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b690f57-5989-4190-a126-4a110320bf43 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.636275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b690f57-5989-4190-a126-4a110320bf43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.638151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b690f57-5989-4190-a126-4a110320bf43 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.649435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 800/1598] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.650355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b690f57-5989-4190-a126-4a110320bf43 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.661301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b690f57-5989-4190-a126-4a110320bf43 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.666777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519478e7-18c2-453d-bea3-584b56e72364 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.667515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519478e7-18c2-453d-bea3-584b56e72364 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.667992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519478e7-18c2-453d-bea3-584b56e72364 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.668350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519478e7-18c2-453d-bea3-584b56e72364 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.669156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519478e7-18c2-453d-bea3-584b56e72364 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.670837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519478e7-18c2-453d-bea3-584b56e72364 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.673070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 799/1599] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:14 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:14.674038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e483802-b9cf-4377-b1c2-978f187b7e7b None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.686491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e483802-b9cf-4377-b1c2-978f187b7e7b None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.686712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 801/1600] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.687487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519478e7-18c2-453d-bea3-584b56e72364 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.694948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff1982c-eb49-4950-8b63-762cea0318b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.695660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1982c-eb49-4950-8b63-762cea0318b2 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.696046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1982c-eb49-4950-8b63-762cea0318b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.696289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1982c-eb49-4950-8b63-762cea0318b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.697100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff1982c-eb49-4950-8b63-762cea0318b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.698711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff1982c-eb49-4950-8b63-762cea0318b2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.699554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519478e7-18c2-453d-bea3-584b56e72364 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.705040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db7e136-9442-435b-971e-4e725cb69e29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.705682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db7e136-9442-435b-971e-4e725cb69e29 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.706045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db7e136-9442-435b-971e-4e725cb69e29 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.706422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db7e136-9442-435b-971e-4e725cb69e29 admin admin] PATH_INFO: `/v3/projects/e5f4896d1efd4372a13dbcaeda54bfe2/users/a76550322ec7408fa63ef23d64267daa/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.710557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 800/1601] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:14.711826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff1982c-eb49-4950-8b63-762cea0318b2 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.722406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff1982c-eb49-4950-8b63-762cea0318b2 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.728456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db7e136-9442-435b-971e-4e725cb69e29 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5f4896d1efd4372a13dbcaeda54bfe2, user_id=a76550322ec7408fa63ef23d64267daa, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.728569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57e5b41-f1a4-4289-ba06-bbb51ba1e40e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.729248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e5b41-f1a4-4289-ba06-bbb51ba1e40e admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.729529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e5b41-f1a4-4289-ba06-bbb51ba1e40e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.729919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57e5b41-f1a4-4289-ba06-bbb51ba1e40e admin admin] PATH_INFO: `/v3/projects/6e872b2ca5744cd9bd3202c992083643/users/d65fb4f666e6485682ff51c3427cf183/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.731896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7db7e136-9442-435b-971e-4e725cb69e29 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.747192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57e5b41-f1a4-4289-ba06-bbb51ba1e40e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e872b2ca5744cd9bd3202c992083643, user_id=d65fb4f666e6485682ff51c3427cf183, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.758974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 802/1602] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:14 2022] PUT /identity/v3/projects/e5f4896d1efd4372a13dbcaeda54bfe2/users/a76550322ec7408fa63ef23d64267daa/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:14.758974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db7e136-9442-435b-971e-4e725cb69e29 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.759295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57e5b41-f1a4-4289-ba06-bbb51ba1e40e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.759402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7db7e136-9442-435b-971e-4e725cb69e29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:14.759496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:14.785292 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 801/1603] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:14 2022] PUT /identity/v3/projects/6e872b2ca5744cd9bd3202c992083643/users/d65fb4f666e6485682ff51c3427cf183/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:14.785292 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57e5b41-f1a4-4289-ba06-bbb51ba1e40e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.846319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db7e136-9442-435b-971e-4e725cb69e29 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.873878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57e5b41-f1a4-4289-ba06-bbb51ba1e40e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:14.879317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468f62bc-fed9-4175-9b93-cd749f7017fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.880003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468f62bc-fed9-4175-9b93-cd749f7017fa admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.880299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468f62bc-fed9-4175-9b93-cd749f7017fa admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.880583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468f62bc-fed9-4175-9b93-cd749f7017fa admin admin] PATH_INFO: `/v3/projects/946bf1e173f9476d86cf6133055fba0a/users/8b7751b93f88424abebe6859f46f8351/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.899142 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468f62bc-fed9-4175-9b93-cd749f7017fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=946bf1e173f9476d86cf6133055fba0a, user_id=8b7751b93f88424abebe6859f46f8351, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.901031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7582c8a-4754-4809-a1d5-49f8065a7cd0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:14.901712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7582c8a-4754-4809-a1d5-49f8065a7cd0 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:14.901994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7582c8a-4754-4809-a1d5-49f8065a7cd0 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:14.902110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468f62bc-fed9-4175-9b93-cd749f7017fa admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.902273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7582c8a-4754-4809-a1d5-49f8065a7cd0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:14.911840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7582c8a-4754-4809-a1d5-49f8065a7cd0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:14.913948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7582c8a-4754-4809-a1d5-49f8065a7cd0 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:14.918229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 802/1604] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:14 2022] PUT /identity/v3/projects/946bf1e173f9476d86cf6133055fba0a/users/8b7751b93f88424abebe6859f46f8351/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:14.919275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468f62bc-fed9-4175-9b93-cd749f7017fa admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.928499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 803/1605] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:14.929671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7582c8a-4754-4809-a1d5-49f8065a7cd0 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:14.940865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7582c8a-4754-4809-a1d5-49f8065a7cd0 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:15.007771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b632579-f6fe-4528-b8a4-1075605df63e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:15.009080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b632579-f6fe-4528-b8a4-1075605df63e admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.009080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b632579-f6fe-4528-b8a4-1075605df63e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.009572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b632579-f6fe-4528-b8a4-1075605df63e admin admin] PATH_INFO: `/v3/projects/84bbb598d5204cad9c8cb378bc09f1c3/users/865e73a677c0412ab5619e92905526dd/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.029637 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b632579-f6fe-4528-b8a4-1075605df63e admin admin] RBAC: Authorizing `identity:create_grant(project_id=84bbb598d5204cad9c8cb378bc09f1c3, user_id=865e73a677c0412ab5619e92905526dd, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:15.034609 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b632579-f6fe-4528-b8a4-1075605df63e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:15.045197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468f62bc-fed9-4175-9b93-cd749f7017fa admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:15.050103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:15.052458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017bfc10-e0a5-45ea-92dc-991ddc7a846d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:15.054481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017bfc10-e0a5-45ea-92dc-991ddc7a846d admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.054782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017bfc10-e0a5-45ea-92dc-991ddc7a846d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.055136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017bfc10-e0a5-45ea-92dc-991ddc7a846d admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.055917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017bfc10-e0a5-45ea-92dc-991ddc7a846d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:15.057960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 804/1606] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:14 2022] PUT /identity/v3/projects/84bbb598d5204cad9c8cb378bc09f1c3/users/865e73a677c0412ab5619e92905526dd/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:15.059712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017bfc10-e0a5-45ea-92dc-991ddc7a846d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:15.069166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b632579-f6fe-4528-b8a4-1075605df63e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:15.098662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 803/1607] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:14 2022] POST /identity/v3/projects => generated 371 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:15.102506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94db9c0-3351-41c8-bbb2-dab87ab1a3e0 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.103224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94db9c0-3351-41c8-bbb2-dab87ab1a3e0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.103758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94db9c0-3351-41c8-bbb2-dab87ab1a3e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.145631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a94db9c0-3351-41c8-bbb2-dab87ab1a3e0 None None] MFA Rules not processed for user `a76550322ec7408fa63ef23d64267daa`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:15.148957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a94db9c0-3351-41c8-bbb2-dab87ab1a3e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:15.153931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b632579-f6fe-4528-b8a4-1075605df63e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:15.235255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:15.243953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260659cf-03f4-470f-b70c-8970aa6e39ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:15.244757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260659cf-03f4-470f-b70c-8970aa6e39ab admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.245246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260659cf-03f4-470f-b70c-8970aa6e39ab admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.245640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260659cf-03f4-470f-b70c-8970aa6e39ab admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.247852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260659cf-03f4-470f-b70c-8970aa6e39ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:15.252410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260659cf-03f4-470f-b70c-8970aa6e39ab admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:15.273105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 805/1608] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:15 2022] GET /identity/v3/roles => generated 1420 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:15.273105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b7578a-be09-4229-a1a1-c49f518d28e1 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.273105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b7578a-be09-4229-a1a1-c49f518d28e1 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.273105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b7578a-be09-4229-a1a1-c49f518d28e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.277510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 804/1609] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:15 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:15.279504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f848f3e5-0bd1-4093-b15c-4c8981661ed1 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.279970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f848f3e5-0bd1-4093-b15c-4c8981661ed1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.280907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f848f3e5-0bd1-4093-b15c-4c8981661ed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.322498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-d9b7578a-be09-4229-a1a1-c49f518d28e1 None None] MFA Rules not processed for user `d65fb4f666e6485682ff51c3427cf183`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:15.325247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d9b7578a-be09-4229-a1a1-c49f518d28e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:15.338298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f848f3e5-0bd1-4093-b15c-4c8981661ed1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:15.341848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f848f3e5-0bd1-4093-b15c-4c8981661ed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:15.371330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 805/1610] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:15.375514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72cc3c3-4e13-42e7-b7e7-1b9ab071c803 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.376440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72cc3c3-4e13-42e7-b7e7-1b9ab071c803 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.376905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72cc3c3-4e13-42e7-b7e7-1b9ab071c803 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.458381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 806/1611] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:15 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:15.458381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b7578a-be09-4229-a1a1-c49f518d28e1 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:15.469844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f72cc3c3-4e13-42e7-b7e7-1b9ab071c803 None None] MFA Rules not processed for user `865e73a677c0412ab5619e92905526dd`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:15.470058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f72cc3c3-4e13-42e7-b7e7-1b9ab071c803 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:15.471745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b7578a-be09-4229-a1a1-c49f518d28e1 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:15.481913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985ea93b-2ec7-4ad8-9540-0e81c4ce3c25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:15.482823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985ea93b-2ec7-4ad8-9540-0e81c4ce3c25 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.483195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985ea93b-2ec7-4ad8-9540-0e81c4ce3c25 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.483871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985ea93b-2ec7-4ad8-9540-0e81c4ce3c25 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.484528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985ea93b-2ec7-4ad8-9540-0e81c4ce3c25 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:15.487093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985ea93b-2ec7-4ad8-9540-0e81c4ce3c25 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:15.531441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 807/1612] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:15 2022] POST /identity/v3/users => generated 292 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:15.532694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985ea93b-2ec7-4ad8-9540-0e81c4ce3c25 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:15.551676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985ea93b-2ec7-4ad8-9540-0e81c4ce3c25 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:15.559926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed5741-d12d-4714-b3f3-2c626696637d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:15.560951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed5741-d12d-4714-b3f3-2c626696637d admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.561660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed5741-d12d-4714-b3f3-2c626696637d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.562164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faed5741-d12d-4714-b3f3-2c626696637d admin admin] PATH_INFO: `/v3/projects/461e2af738084ccb986169d752510588/users/572408fd157844dcae13000d8e9c4b7a/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.580921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:15.587060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faed5741-d12d-4714-b3f3-2c626696637d admin admin] RBAC: Authorizing `identity:create_grant(project_id=461e2af738084ccb986169d752510588, user_id=572408fd157844dcae13000d8e9c4b7a, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:15.591698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faed5741-d12d-4714-b3f3-2c626696637d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:15.623317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 808/1613] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:15 2022] PUT /identity/v3/projects/461e2af738084ccb986169d752510588/users/572408fd157844dcae13000d8e9c4b7a/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:15.627229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed5741-d12d-4714-b3f3-2c626696637d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:15.689327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 806/1614] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:15.698303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72cc3c3-4e13-42e7-b7e7-1b9ab071c803 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:15.767284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faed5741-d12d-4714-b3f3-2c626696637d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:15.775317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a337dcd-7f7c-4809-b678-a7c4e757a385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:15.775962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a337dcd-7f7c-4809-b678-a7c4e757a385 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.776722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a337dcd-7f7c-4809-b678-a7c4e757a385 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.776722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a337dcd-7f7c-4809-b678-a7c4e757a385 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.777238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a337dcd-7f7c-4809-b678-a7c4e757a385 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:15.780713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a337dcd-7f7c-4809-b678-a7c4e757a385 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:15.830864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72cc3c3-4e13-42e7-b7e7-1b9ab071c803 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:15.854011 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 809/1615] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:15 2022] POST /identity/v3/projects => generated 373 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:15.855133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a337dcd-7f7c-4809-b678-a7c4e757a385 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:15.964264 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee7c350-5066-4514-b17d-e3033c1010f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:15.966283 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee7c350-5066-4514-b17d-e3033c1010f3 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:15.967048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee7c350-5066-4514-b17d-e3033c1010f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:15.967405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee7c350-5066-4514-b17d-e3033c1010f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:15.985779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee7c350-5066-4514-b17d-e3033c1010f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:15.992467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee7c350-5066-4514-b17d-e3033c1010f3 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:15.999610 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a337dcd-7f7c-4809-b678-a7c4e757a385 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.015609 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d78008e-c419-45a9-aee6-bb604ab0d8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.016586 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d78008e-c419-45a9-aee6-bb604ab0d8a1 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.016904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d78008e-c419-45a9-aee6-bb604ab0d8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.017190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d78008e-c419-45a9-aee6-bb604ab0d8a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.018228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d78008e-c419-45a9-aee6-bb604ab0d8a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.022698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d78008e-c419-45a9-aee6-bb604ab0d8a1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.036371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 807/1616] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:16.037862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee7c350-5066-4514-b17d-e3033c1010f3 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.065319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee7c350-5066-4514-b17d-e3033c1010f3 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.079357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 810/1617] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:15 2022] POST /identity/v3/projects => generated 385 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:16.081376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e186e1c7-9f27-4a49-b492-13a5213b949a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.081805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c23d39-b651-41c8-b12f-94d719d086ae None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.082020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e186e1c7-9f27-4a49-b492-13a5213b949a admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.082313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e186e1c7-9f27-4a49-b492-13a5213b949a admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.082596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e186e1c7-9f27-4a49-b492-13a5213b949a admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.082841 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c23d39-b651-41c8-b12f-94d719d086ae None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.083476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c23d39-b651-41c8-b12f-94d719d086ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.087671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e186e1c7-9f27-4a49-b492-13a5213b949a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.093755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e186e1c7-9f27-4a49-b492-13a5213b949a admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.116408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-d4c23d39-b651-41c8-b12f-94d719d086ae None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:16.121331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d4c23d39-b651-41c8-b12f-94d719d086ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:16.131314 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 808/1618] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:16 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:16.135102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef58883a-287e-478c-b831-8aedc6365f17 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.135411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef58883a-287e-478c-b831-8aedc6365f17 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.135753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef58883a-287e-478c-b831-8aedc6365f17 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.144552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 809/1619] 10.222.0.40 () {54 vars in 962 bytes} [Fri Jul 29 04:47:16 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:47:16.146267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef58883a-287e-478c-b831-8aedc6365f17 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.252786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:16.254206 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 811/1620] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:16.258596 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c23d39-b651-41c8-b12f-94d719d086ae None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.269180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:16.282485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c23d39-b651-41c8-b12f-94d719d086ae None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.290809 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f6a85c-3d3e-447b-8108-7a77eb416570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.292549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f6a85c-3d3e-447b-8108-7a77eb416570 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.292977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f6a85c-3d3e-447b-8108-7a77eb416570 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.293416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f6a85c-3d3e-447b-8108-7a77eb416570 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.294209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f6a85c-3d3e-447b-8108-7a77eb416570 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.301280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f6a85c-3d3e-447b-8108-7a77eb416570 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.345779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef58883a-287e-478c-b831-8aedc6365f17 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.349485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 812/1621] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:16 2022] POST /identity/v3/users => generated 299 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:16.350588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f6a85c-3d3e-447b-8108-7a77eb416570 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.352397 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63184644-2726-4134-adfb-6d778711b957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.353164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63184644-2726-4134-adfb-6d778711b957 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.353511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63184644-2726-4134-adfb-6d778711b957 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.353798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63184644-2726-4134-adfb-6d778711b957 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.355111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63184644-2726-4134-adfb-6d778711b957 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.360634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63184644-2726-4134-adfb-6d778711b957 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.377673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f6a85c-3d3e-447b-8108-7a77eb416570 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.392224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df23f11-cf59-4c43-8c11-0b43a14d472f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.392224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df23f11-cf59-4c43-8c11-0b43a14d472f admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.392224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df23f11-cf59-4c43-8c11-0b43a14d472f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.392685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df23f11-cf59-4c43-8c11-0b43a14d472f admin admin] PATH_INFO: `/v3/projects/3ba444c7610642f09eb0e5ce86465768/users/260125e92698404b96f344ae46d34083/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.410147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 810/1622] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:16 2022] POST /identity/v3/users => generated 293 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:16.410387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921c7c36-e92f-4abc-9f5f-5a6fd33d76b5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.410570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921c7c36-e92f-4abc-9f5f-5a6fd33d76b5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.410860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921c7c36-e92f-4abc-9f5f-5a6fd33d76b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.429776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df23f11-cf59-4c43-8c11-0b43a14d472f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ba444c7610642f09eb0e5ce86465768, user_id=260125e92698404b96f344ae46d34083, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.436831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df23f11-cf59-4c43-8c11-0b43a14d472f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.460379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 813/1623] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:16 2022] PUT /identity/v3/projects/3ba444c7610642f09eb0e5ce86465768/users/260125e92698404b96f344ae46d34083/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:16.461336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df23f11-cf59-4c43-8c11-0b43a14d472f admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.506893 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-921c7c36-e92f-4abc-9f5f-5a6fd33d76b5 None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:16.507955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-921c7c36-e92f-4abc-9f5f-5a6fd33d76b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:16.534500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df23f11-cf59-4c43-8c11-0b43a14d472f admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.543101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b295681b-3877-4267-af64-df83d220c91c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.543842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b295681b-3877-4267-af64-df83d220c91c admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.544136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b295681b-3877-4267-af64-df83d220c91c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.544419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b295681b-3877-4267-af64-df83d220c91c admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.545723 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b295681b-3877-4267-af64-df83d220c91c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.548665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b295681b-3877-4267-af64-df83d220c91c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.566135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 814/1624] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:16 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:16.567575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b295681b-3877-4267-af64-df83d220c91c admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.658297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b295681b-3877-4267-af64-df83d220c91c admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.663510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede51988-975b-4b7b-a92f-74ce71519308 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.664340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede51988-975b-4b7b-a92f-74ce71519308 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.664884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede51988-975b-4b7b-a92f-74ce71519308 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.665319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede51988-975b-4b7b-a92f-74ce71519308 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.667864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 811/1625] 10.222.0.40 () {58 vars in 1059 bytes} [Fri Jul 29 04:47:16 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:16.669017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921c7c36-e92f-4abc-9f5f-5a6fd33d76b5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.677800 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede51988-975b-4b7b-a92f-74ce71519308 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.679588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ede51988-975b-4b7b-a92f-74ce71519308 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.696335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921c7c36-e92f-4abc-9f5f-5a6fd33d76b5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.696994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 815/1626] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:16.698186 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede51988-975b-4b7b-a92f-74ce71519308 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.703005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a27cb-6fd8-4789-9b4f-7cbfbcc47d27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.703665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a27cb-6fd8-4789-9b4f-7cbfbcc47d27 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.703961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a27cb-6fd8-4789-9b4f-7cbfbcc47d27 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.704239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6a27cb-6fd8-4789-9b4f-7cbfbcc47d27 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.704993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a27cb-6fd8-4789-9b4f-7cbfbcc47d27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.706705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6a27cb-6fd8-4789-9b4f-7cbfbcc47d27 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.711299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede51988-975b-4b7b-a92f-74ce71519308 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.732326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 812/1627] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:16 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:16.733852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a27cb-6fd8-4789-9b4f-7cbfbcc47d27 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.734605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a56e2ca-a3d9-4fe7-8b62-a7f7333e3dc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.735522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a56e2ca-a3d9-4fe7-8b62-a7f7333e3dc1 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.736404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a56e2ca-a3d9-4fe7-8b62-a7f7333e3dc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.741195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a56e2ca-a3d9-4fe7-8b62-a7f7333e3dc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.744053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a56e2ca-a3d9-4fe7-8b62-a7f7333e3dc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.746238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a56e2ca-a3d9-4fe7-8b62-a7f7333e3dc1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.758999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6a27cb-6fd8-4789-9b4f-7cbfbcc47d27 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.767751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c50f68-20bd-46ef-8075-a359314f7a69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.768770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c50f68-20bd-46ef-8075-a359314f7a69 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.768770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c50f68-20bd-46ef-8075-a359314f7a69 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.768977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c50f68-20bd-46ef-8075-a359314f7a69 admin admin] PATH_INFO: `/v3/projects/169b869b7d184582aadc6ddb2748fa24/users/1ad793f84a334458aead6ec066bf4010/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.772860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 816/1628] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:16 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:16.774260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a56e2ca-a3d9-4fe7-8b62-a7f7333e3dc1 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.796897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a56e2ca-a3d9-4fe7-8b62-a7f7333e3dc1 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.805233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59071b-b9b1-4cb8-9e9a-f00105f6f302 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:16.806388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59071b-b9b1-4cb8-9e9a-f00105f6f302 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:16.806883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59071b-b9b1-4cb8-9e9a-f00105f6f302 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:16.807371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59071b-b9b1-4cb8-9e9a-f00105f6f302 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:16.811425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c50f68-20bd-46ef-8075-a359314f7a69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=169b869b7d184582aadc6ddb2748fa24, user_id=1ad793f84a334458aead6ec066bf4010, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.814269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c50f68-20bd-46ef-8075-a359314f7a69 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.842344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 813/1629] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:16 2022] PUT /identity/v3/projects/169b869b7d184582aadc6ddb2748fa24/users/1ad793f84a334458aead6ec066bf4010/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:16.842966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c50f68-20bd-46ef-8075-a359314f7a69 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.870480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:16.887650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a59071b-b9b1-4cb8-9e9a-f00105f6f302 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:16.892341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a59071b-b9b1-4cb8-9e9a-f00105f6f302 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:16.923844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 817/1630] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:16 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:16.926325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59071b-b9b1-4cb8-9e9a-f00105f6f302 service nova] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:16.964761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59071b-b9b1-4cb8-9e9a-f00105f6f302 service nova] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:16.980345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:17.001373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c50f68-20bd-46ef-8075-a359314f7a69 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:17.016390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f90f0-8708-4448-ac6a-3709218c1e31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:17.019163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f90f0-8708-4448-ac6a-3709218c1e31 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.020651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f90f0-8708-4448-ac6a-3709218c1e31 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.021539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f90f0-8708-4448-ac6a-3709218c1e31 admin admin] PATH_INFO: `/v3/projects/5d3a871b601341a4a4d3d3b5bbd27b5b/users/5e609c41ba94499285150cb9471e16a7/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.071048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f90f0-8708-4448-ac6a-3709218c1e31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d3a871b601341a4a4d3d3b5bbd27b5b, user_id=5e609c41ba94499285150cb9471e16a7, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:17.077936 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f90f0-8708-4448-ac6a-3709218c1e31 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:17.101637 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994e41c6-09cb-4cad-a3b5-d040738212b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:17.105685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994e41c6-09cb-4cad-a3b5-d040738212b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.107213 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994e41c6-09cb-4cad-a3b5-d040738212b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.107660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994e41c6-09cb-4cad-a3b5-d040738212b7 admin admin] PATH_INFO: `/v3/projects/3ba444c7610642f09eb0e5ce86465768/users/260125e92698404b96f344ae46d34083/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.109260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 814/1631] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:16 2022] PUT /identity/v3/projects/5d3a871b601341a4a4d3d3b5bbd27b5b/users/5e609c41ba94499285150cb9471e16a7/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:17.111072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f90f0-8708-4448-ac6a-3709218c1e31 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:17.116429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994e41c6-09cb-4cad-a3b5-d040738212b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ba444c7610642f09eb0e5ce86465768, user_id=260125e92698404b96f344ae46d34083, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:17.122106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994e41c6-09cb-4cad-a3b5-d040738212b7 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:17.152674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 818/1632] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:16 2022] PUT /identity/v3/projects/3ba444c7610642f09eb0e5ce86465768/users/260125e92698404b96f344ae46d34083/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:17.157719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994e41c6-09cb-4cad-a3b5-d040738212b7 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:17.278297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f90f0-8708-4448-ac6a-3709218c1e31 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:17.320365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994e41c6-09cb-4cad-a3b5-d040738212b7 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:17.334983 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa40d90-16ab-47b9-beab-d90c3c15f139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:17.335857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa40d90-16ab-47b9-beab-d90c3c15f139 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.336134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa40d90-16ab-47b9-beab-d90c3c15f139 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.336456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa40d90-16ab-47b9-beab-d90c3c15f139 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.337204 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa40d90-16ab-47b9-beab-d90c3c15f139 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:17.339030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa40d90-16ab-47b9-beab-d90c3c15f139 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:17.358166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe75128-df99-4952-9e5a-6db3eabca1fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:17.359048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe75128-df99-4952-9e5a-6db3eabca1fe admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.359382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe75128-df99-4952-9e5a-6db3eabca1fe admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.359905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe75128-df99-4952-9e5a-6db3eabca1fe admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.361553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe75128-df99-4952-9e5a-6db3eabca1fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:17.363470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe75128-df99-4952-9e5a-6db3eabca1fe admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:17.369544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 819/1633] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:17 2022] GET /identity/v3/roles => generated 1420 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:17.371034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9901dc2c-529d-4da0-b584-e65fc303a0c5 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.371331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9901dc2c-529d-4da0-b584-e65fc303a0c5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.371611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9901dc2c-529d-4da0-b584-e65fc303a0c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.379330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 815/1634] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:17 2022] GET /identity/v3/roles => generated 1420 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:17.381051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5399de00-1eb9-4e70-83ed-dbaa378c7d55 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.381415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5399de00-1eb9-4e70-83ed-dbaa378c7d55 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.381824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5399de00-1eb9-4e70-83ed-dbaa378c7d55 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.389697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 816/1635] 10.222.0.40 () {54 vars in 912 bytes} [Fri Jul 29 04:47:17 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:47:17.391160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5399de00-1eb9-4e70-83ed-dbaa378c7d55 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:17.408979 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5399de00-1eb9-4e70-83ed-dbaa378c7d55 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:17.415118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb4d6aa-31f8-402b-a73e-2fffd4a315e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:17.416184 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb4d6aa-31f8-402b-a73e-2fffd4a315e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.416450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb4d6aa-31f8-402b-a73e-2fffd4a315e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.416895 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb4d6aa-31f8-402b-a73e-2fffd4a315e4 admin admin] PATH_INFO: `/v3/projects/5d3a871b601341a4a4d3d3b5bbd27b5b/users/5e609c41ba94499285150cb9471e16a7/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.419977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb4d6aa-31f8-402b-a73e-2fffd4a315e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d3a871b601341a4a4d3d3b5bbd27b5b, user_id=5e609c41ba94499285150cb9471e16a7, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:17.423090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb4d6aa-31f8-402b-a73e-2fffd4a315e4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:17.437293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-9901dc2c-529d-4da0-b584-e65fc303a0c5 None None] MFA Rules not processed for user `260125e92698404b96f344ae46d34083`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:17.438545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9901dc2c-529d-4da0-b584-e65fc303a0c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:17.446154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 817/1636] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:17 2022] PUT /identity/v3/projects/5d3a871b601341a4a4d3d3b5bbd27b5b/users/5e609c41ba94499285150cb9471e16a7/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:17.447304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb4d6aa-31f8-402b-a73e-2fffd4a315e4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:17.531577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb4d6aa-31f8-402b-a73e-2fffd4a315e4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:17.537414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81bfd32d-9063-4be5-8386-e427891b0100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:17.546290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bfd32d-9063-4be5-8386-e427891b0100 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.547032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bfd32d-9063-4be5-8386-e427891b0100 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.547384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81bfd32d-9063-4be5-8386-e427891b0100 admin admin] PATH_INFO: `/v3/projects/169b869b7d184582aadc6ddb2748fa24/users/1ad793f84a334458aead6ec066bf4010/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.551077 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bfd32d-9063-4be5-8386-e427891b0100 admin admin] RBAC: Authorizing `identity:create_grant(project_id=169b869b7d184582aadc6ddb2748fa24, user_id=1ad793f84a334458aead6ec066bf4010, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:17.558645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81bfd32d-9063-4be5-8386-e427891b0100 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:17.573151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 818/1637] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:17 2022] PUT /identity/v3/projects/169b869b7d184582aadc6ddb2748fa24/users/1ad793f84a334458aead6ec066bf4010/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:17.574725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c65d143-240e-4ecd-b284-cab17b12abed None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.575122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c65d143-240e-4ecd-b284-cab17b12abed None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.575501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c65d143-240e-4ecd-b284-cab17b12abed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.591818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 820/1638] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:17 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:17.600250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:17.607569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90011f3a-cd38-4a41-8c44-f76a208cbbae None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.607985 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90011f3a-cd38-4a41-8c44-f76a208cbbae None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.608276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90011f3a-cd38-4a41-8c44-f76a208cbbae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.616561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7c65d143-240e-4ecd-b284-cab17b12abed None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:17.618042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7c65d143-240e-4ecd-b284-cab17b12abed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:17.629236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:17.674858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-90011f3a-cd38-4a41-8c44-f76a208cbbae None None] MFA Rules not processed for user `5e609c41ba94499285150cb9471e16a7`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:17.676363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-90011f3a-cd38-4a41-8c44-f76a208cbbae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:17.722187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 819/1639] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:47:17 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:17.730276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80df87bf-8334-4f32-8273-1caf3851a138 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:17.730577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80df87bf-8334-4f32-8273-1caf3851a138 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:17.731227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80df87bf-8334-4f32-8273-1caf3851a138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:17.797749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-80df87bf-8334-4f32-8273-1caf3851a138 None None] MFA Rules not processed for user `1ad793f84a334458aead6ec066bf4010`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:17.801663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-80df87bf-8334-4f32-8273-1caf3851a138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:17.871998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 821/1640] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:17 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:17.873465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90011f3a-cd38-4a41-8c44-f76a208cbbae None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:17.950563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 820/1641] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:17 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:17.951784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80df87bf-8334-4f32-8273-1caf3851a138 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:18.049239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90011f3a-cd38-4a41-8c44-f76a208cbbae None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:18.051670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa591a31-0433-4d7a-96f2-700bfd21b014 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:18.053329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa591a31-0433-4d7a-96f2-700bfd21b014 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:18.054614 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa591a31-0433-4d7a-96f2-700bfd21b014 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:18.055176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa591a31-0433-4d7a-96f2-700bfd21b014 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:18.059187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80df87bf-8334-4f32-8273-1caf3851a138 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:18.076696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34df1784-ce3e-4baf-88c3-0cfed2d7286b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:18.077350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34df1784-ce3e-4baf-88c3-0cfed2d7286b service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:18.077634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34df1784-ce3e-4baf-88c3-0cfed2d7286b service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:18.077908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34df1784-ce3e-4baf-88c3-0cfed2d7286b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:18.093424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34df1784-ce3e-4baf-88c3-0cfed2d7286b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:18.095051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34df1784-ce3e-4baf-88c3-0cfed2d7286b service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:18.115027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 821/1642] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:17 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:18.207850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa591a31-0433-4d7a-96f2-700bfd21b014 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:18.209791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa591a31-0433-4d7a-96f2-700bfd21b014 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:18.222989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 822/1643] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:17 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:18.360950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34df1784-ce3e-4baf-88c3-0cfed2d7286b service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:18.380669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34df1784-ce3e-4baf-88c3-0cfed2d7286b service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:18.389557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f4720c-97c9-4904-ad6d-02fc4fad9e36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:18.390602 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f4720c-97c9-4904-ad6d-02fc4fad9e36 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:18.391070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f4720c-97c9-4904-ad6d-02fc4fad9e36 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:18.391556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f4720c-97c9-4904-ad6d-02fc4fad9e36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:18.427394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f4720c-97c9-4904-ad6d-02fc4fad9e36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:18.432977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f4720c-97c9-4904-ad6d-02fc4fad9e36 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:18.478242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 822/1644] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:18 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:18.521159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa591a31-0433-4d7a-96f2-700bfd21b014 service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:18.532277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa591a31-0433-4d7a-96f2-700bfd21b014 service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:18.538982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f39f8a2-51b8-4c3e-8eb6-20bf57144d15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:18.540053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f39f8a2-51b8-4c3e-8eb6-20bf57144d15 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:18.540555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f39f8a2-51b8-4c3e-8eb6-20bf57144d15 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:18.541110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f39f8a2-51b8-4c3e-8eb6-20bf57144d15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:18.556803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f39f8a2-51b8-4c3e-8eb6-20bf57144d15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:18.558702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f39f8a2-51b8-4c3e-8eb6-20bf57144d15 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:18.585300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 823/1645] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:18 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:18.842106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f4720c-97c9-4904-ad6d-02fc4fad9e36 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:18.863970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:18.985281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f39f8a2-51b8-4c3e-8eb6-20bf57144d15 service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:18.995273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f39f8a2-51b8-4c3e-8eb6-20bf57144d15 service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:19.010859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80d6a05-bc22-48f5-b5a1-c1520ac1b1b7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:19.012468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80d6a05-bc22-48f5-b5a1-c1520ac1b1b7 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:19.012786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80d6a05-bc22-48f5-b5a1-c1520ac1b1b7 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:19.013069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80d6a05-bc22-48f5-b5a1-c1520ac1b1b7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:19.050267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f4720c-97c9-4904-ad6d-02fc4fad9e36 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:19.067423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:19.068107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:19.068406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:19.073717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] PATH_INFO: `/v3/users/d65fb4f666e6485682ff51c3427cf183` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:19.078432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d65fb4f666e6485682ff51c3427cf183)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:19.086265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: WARNING py.warnings [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:47:19.086265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: warnings.warn(msg) Jul 29 04:47:19.086265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: Jul 29 04:47:19.091495 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:19.113122 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:19.257145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80d6a05-bc22-48f5-b5a1-c1520ac1b1b7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:19.258640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80d6a05-bc22-48f5-b5a1-c1520ac1b1b7 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:19.305910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 824/1646] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:18 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:47:19.316464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd65fb4f666e6485682ff51c3427cf183'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:19.317242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd65fb4f666e6485682ff51c3427cf183'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:19.318140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd65fb4f666e6485682ff51c3427cf183'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:19.375988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd65fb4f666e6485682ff51c3427cf183'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:19.402947 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd65fb4f666e6485682ff51c3427cf183'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:19.431677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 823/1647] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:18 2022] DELETE /identity/v3/users/d65fb4f666e6485682ff51c3427cf183 => generated 0 bytes in 595 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:19.436005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c81edd-46d7-4ba2-88a4-9a05e9f46b30 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:19.436297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c81edd-46d7-4ba2-88a4-9a05e9f46b30 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:19.436581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c81edd-46d7-4ba2-88a4-9a05e9f46b30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:19.484968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-09c81edd-46d7-4ba2-88a4-9a05e9f46b30 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:19.486142 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-09c81edd-46d7-4ba2-88a4-9a05e9f46b30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:19.679183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 825/1648] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:19.797505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:19.799117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:19.990757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a2e87e-0e3e-4dac-a136-ff3a4bbe2468 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:20.002228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3d1fab-02cc-4ad7-844e-4ea76cd47417 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:20.003133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3d1fab-02cc-4ad7-844e-4ea76cd47417 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:20.003715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3d1fab-02cc-4ad7-844e-4ea76cd47417 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:20.004522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3d1fab-02cc-4ad7-844e-4ea76cd47417 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:20.043503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3a5eae-32cb-4b20-a71c-3cbfedfe911c None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:20.043790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3a5eae-32cb-4b20-a71c-3cbfedfe911c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:20.044641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3a5eae-32cb-4b20-a71c-3cbfedfe911c None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:20.046840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 826/1649] 10.222.0.40 () {54 vars in 962 bytes} [Fri Jul 29 04:47:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:47:20.056874 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a859fc02-5273-48f6-8b68-affa25b4bf83 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:20.057587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a859fc02-5273-48f6-8b68-affa25b4bf83 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:20.058017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a859fc02-5273-48f6-8b68-affa25b4bf83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:20.064146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3d1fab-02cc-4ad7-844e-4ea76cd47417 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:20.073168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3d1fab-02cc-4ad7-844e-4ea76cd47417 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:20.102396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 824/1650] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:20.103821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-a859fc02-5273-48f6-8b68-affa25b4bf83 None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:20.105645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a859fc02-5273-48f6-8b68-affa25b4bf83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:20.111052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3d1fab-02cc-4ad7-844e-4ea76cd47417 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:20.113624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ad3d1fab-02cc-4ad7-844e-4ea76cd47417 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:20.145310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3d1fab-02cc-4ad7-844e-4ea76cd47417 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:20.181273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:20.181273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:20.181273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:20.181273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] PATH_INFO: `/v3/users/00463dc0723f414fbd55018d79febc66` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:20.183933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] RBAC: Authorizing `identity:delete_user(user_id=00463dc0723f414fbd55018d79febc66)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:20.183933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:20.268630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:20.302311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '00463dc0723f414fbd55018d79febc66'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:20.304770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '00463dc0723f414fbd55018d79febc66'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:20.306784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '00463dc0723f414fbd55018d79febc66'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:20.315147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 827/1651] 10.222.0.40 () {58 vars in 1059 bytes} [Fri Jul 29 04:47:20 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:20.325692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be326da7-2965-4ef0-ad5c-23ae950cdac2 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:20.326210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be326da7-2965-4ef0-ad5c-23ae950cdac2 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:20.326652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be326da7-2965-4ef0-ad5c-23ae950cdac2 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:20.328474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 828/1652] 10.222.0.40 () {54 vars in 912 bytes} [Fri Jul 29 04:47:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:47:20.333633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c808af23-f774-463a-b7ae-7dbb64df0d16 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:20.335488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c808af23-f774-463a-b7ae-7dbb64df0d16 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:20.335908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c808af23-f774-463a-b7ae-7dbb64df0d16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:20.344972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '00463dc0723f414fbd55018d79febc66'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:20.362495 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-47a3851b-8019-4f7b-930f-79445c42616e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '00463dc0723f414fbd55018d79febc66'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:20.378090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c808af23-f774-463a-b7ae-7dbb64df0d16 None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:20.379389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c808af23-f774-463a-b7ae-7dbb64df0d16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:20.391070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 825/1653] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:20 2022] DELETE /identity/v3/users/00463dc0723f414fbd55018d79febc66 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:47:20.395521 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9492f-cf08-4b90-9973-f8f965135948 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:20.395794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9492f-cf08-4b90-9973-f8f965135948 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:20.396011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9492f-cf08-4b90-9973-f8f965135948 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:20.407905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 829/1654] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:47:20 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:20.419844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c808af23-f774-463a-b7ae-7dbb64df0d16 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:20.421831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c808af23-f774-463a-b7ae-7dbb64df0d16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:20.442821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-afd9492f-cf08-4b90-9973-f8f965135948 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:20.447224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-afd9492f-cf08-4b90-9973-f8f965135948 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:20.541509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:20.548041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c808af23-f774-463a-b7ae-7dbb64df0d16 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:20.591664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 826/1655] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:20.627333 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9492f-cf08-4b90-9973-f8f965135948 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:20.629331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-afd9492f-cf08-4b90-9973-f8f965135948 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:20.654983 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d954e9e-b8f7-4b27-a5b6-0950c67b7f9a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:20.656358 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d954e9e-b8f7-4b27-a5b6-0950c67b7f9a service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:20.656724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d954e9e-b8f7-4b27-a5b6-0950c67b7f9a service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:20.657011 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d954e9e-b8f7-4b27-a5b6-0950c67b7f9a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:20.787602 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9492f-cf08-4b90-9973-f8f965135948 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:20.797311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5104c14b-6c37-4c4f-b13a-630229843a93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:20.798146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5104c14b-6c37-4c4f-b13a-630229843a93 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:20.798579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5104c14b-6c37-4c4f-b13a-630229843a93 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:20.799038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5104c14b-6c37-4c4f-b13a-630229843a93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:20.823531 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5104c14b-6c37-4c4f-b13a-630229843a93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:20.825218 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5104c14b-6c37-4c4f-b13a-630229843a93 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:20.844526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 827/1656] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:20.847547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d954e9e-b8f7-4b27-a5b6-0950c67b7f9a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:20.849159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d954e9e-b8f7-4b27-a5b6-0950c67b7f9a service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:20.901876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 830/1657] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:20 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 482 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:21.665979 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5104c14b-6c37-4c4f-b13a-630229843a93 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:21.670597 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5104c14b-6c37-4c4f-b13a-630229843a93 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:21.711675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5104c14b-6c37-4c4f-b13a-630229843a93 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:21.736855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:21.739614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:21.742430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:21.742966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] PATH_INFO: `/v3/projects/6e872b2ca5744cd9bd3202c992083643` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:21.749695 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e872b2ca5744cd9bd3202c992083643)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:21.751956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:21.894239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e872b2ca5744cd9bd3202c992083643', 'request_id': 'req-8876a90a-d385-400b-8faa-c88ab6e28601'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:21.895392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:21.913807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e872b2ca5744cd9bd3202c992083643', 'request_id': 'req-8876a90a-d385-400b-8faa-c88ab6e28601'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:21.925535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e872b2ca5744cd9bd3202c992083643', 'request_id': 'req-8876a90a-d385-400b-8faa-c88ab6e28601'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:21.944653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] The token cache is being invalidate because project 6e872b2ca5744cd9bd3202c992083643 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:21.944960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:21.950099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:21.952117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 828/1658] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:21 2022] DELETE /identity/v3/projects/6e872b2ca5744cd9bd3202c992083643 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:21.956220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d954e9e-b8f7-4b27-a5b6-0950c67b7f9a service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:21.996869 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d954e9e-b8f7-4b27-a5b6-0950c67b7f9a service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:22.129747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:22.130483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:22.132417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:22.132851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] PATH_INFO: `/v3/users/1ad793f84a334458aead6ec066bf4010` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:22.134899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ad793f84a334458aead6ec066bf4010)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:22.140653 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:22.253550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ad793f84a334458aead6ec066bf4010'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:22.257364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:22.272760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ad793f84a334458aead6ec066bf4010'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:22.273102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ad793f84a334458aead6ec066bf4010'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:22.273793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ad793f84a334458aead6ec066bf4010'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:22.287876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ad793f84a334458aead6ec066bf4010'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:22.328857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 831/1659] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:21 2022] DELETE /identity/v3/users/1ad793f84a334458aead6ec066bf4010 => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:22.791552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:22.794343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:22.818013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8876a90a-d385-400b-8faa-c88ab6e28601 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:22.956057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:22.961420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:22.961972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:22.962466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] PATH_INFO: `/v3/projects/2cd7d10d7e2a42ca9afa3ea77a420848` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:22.964120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2cd7d10d7e2a42ca9afa3ea77a420848)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:22.966579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:23.093592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2cd7d10d7e2a42ca9afa3ea77a420848', 'request_id': 'req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:23.108707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2cd7d10d7e2a42ca9afa3ea77a420848', 'request_id': 'req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:23.132235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2cd7d10d7e2a42ca9afa3ea77a420848', 'request_id': 'req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:23.159625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] The token cache is being invalidate because project 2cd7d10d7e2a42ca9afa3ea77a420848 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:23.160033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:23.211984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:23.222489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 829/1660] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:22 2022] DELETE /identity/v3/projects/2cd7d10d7e2a42ca9afa3ea77a420848 => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:23.227319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:23.229701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:23.276151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7b2694-1005-45d7-83cf-d6f86675fa4d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:23.406783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:23.407782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:23.409159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:23.409792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] PATH_INFO: `/v3/users/260125e92698404b96f344ae46d34083` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:23.415015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] RBAC: Authorizing `identity:delete_user(user_id=260125e92698404b96f344ae46d34083)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:23.418029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:23.542077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '260125e92698404b96f344ae46d34083'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:23.558272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '260125e92698404b96f344ae46d34083'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:23.559266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '260125e92698404b96f344ae46d34083'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:23.560039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '260125e92698404b96f344ae46d34083'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:23.575822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-96485992-9ee7-486d-b64d-3992b1cb493b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '260125e92698404b96f344ae46d34083'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:23.617396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 832/1661] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:23 2022] DELETE /identity/v3/users/260125e92698404b96f344ae46d34083 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:24.329779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:24.332252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:24.372220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89da3de4-b659-4fb0-afa0-9a5e6a8bd57c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:24.494153 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:24.496045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:24.496045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:24.496045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] PATH_INFO: `/v3/projects/169b869b7d184582aadc6ddb2748fa24` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:24.497178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] RBAC: Authorizing `identity:delete_project(project_id=169b869b7d184582aadc6ddb2748fa24)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:24.499380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:24.501115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:24.602266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '169b869b7d184582aadc6ddb2748fa24', 'request_id': 'req-451a19e4-c8a9-4817-9f1f-e03833c6c41e'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:24.611328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '169b869b7d184582aadc6ddb2748fa24', 'request_id': 'req-451a19e4-c8a9-4817-9f1f-e03833c6c41e'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:24.624312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '169b869b7d184582aadc6ddb2748fa24', 'request_id': 'req-451a19e4-c8a9-4817-9f1f-e03833c6c41e'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:24.656834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] The token cache is being invalidate because project 169b869b7d184582aadc6ddb2748fa24 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:24.658773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:24.667740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-451a19e4-c8a9-4817-9f1f-e03833c6c41e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:24.685285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 830/1662] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:24 2022] DELETE /identity/v3/projects/169b869b7d184582aadc6ddb2748fa24 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:24.686257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2487a22-5c11-45ef-965a-eee9b5cb9f0a None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:24.687222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2487a22-5c11-45ef-965a-eee9b5cb9f0a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:24.691787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2487a22-5c11-45ef-965a-eee9b5cb9f0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:24.724155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c2487a22-5c11-45ef-965a-eee9b5cb9f0a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:24.726207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c2487a22-5c11-45ef-965a-eee9b5cb9f0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:24.815703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:24.864849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fbfaef-72c5-422e-bb5b-536679b72f3e None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:24.865311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 833/1663] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:24.865655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fbfaef-72c5-422e-bb5b-536679b72f3e None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:24.865940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fbfaef-72c5-422e-bb5b-536679b72f3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:24.905523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-27fbfaef-72c5-422e-bb5b-536679b72f3e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:24.907613 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-27fbfaef-72c5-422e-bb5b-536679b72f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:24.929608 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2487a22-5c11-45ef-965a-eee9b5cb9f0a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:24.937189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 831/1664] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:24.950950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2487a22-5c11-45ef-965a-eee9b5cb9f0a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:24.968961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d549eded-f495-44fc-b8cb-02120bfecfd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:24.969967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d549eded-f495-44fc-b8cb-02120bfecfd2 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:24.970459 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d549eded-f495-44fc-b8cb-02120bfecfd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:24.971036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d549eded-f495-44fc-b8cb-02120bfecfd2 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:24.972921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d549eded-f495-44fc-b8cb-02120bfecfd2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:24.973489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d549eded-f495-44fc-b8cb-02120bfecfd2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:24.977546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d549eded-f495-44fc-b8cb-02120bfecfd2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:25.008616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fbfaef-72c5-422e-bb5b-536679b72f3e None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:25.020352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fbfaef-72c5-422e-bb5b-536679b72f3e None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:25.020649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 834/1665] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:47:25.026765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd1dfe0-c69e-49d1-bef0-40b45948494f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:25.028027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd1dfe0-c69e-49d1-bef0-40b45948494f admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:25.031900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd1dfe0-c69e-49d1-bef0-40b45948494f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:25.032328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd1dfe0-c69e-49d1-bef0-40b45948494f admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:25.033216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd1dfe0-c69e-49d1-bef0-40b45948494f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:25.036587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd1dfe0-c69e-49d1-bef0-40b45948494f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:25.094889 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d549eded-f495-44fc-b8cb-02120bfecfd2 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:25.105436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 832/1666] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:25 2022] POST /identity/v3/projects => generated 367 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:25.111867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d549eded-f495-44fc-b8cb-02120bfecfd2 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:25.231325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a44e00-cdb9-466b-9ffe-03e1a0b990f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:25.233770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a44e00-cdb9-466b-9ffe-03e1a0b990f5 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:25.234078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a44e00-cdb9-466b-9ffe-03e1a0b990f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:25.234423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a44e00-cdb9-466b-9ffe-03e1a0b990f5 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:25.235119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a44e00-cdb9-466b-9ffe-03e1a0b990f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:25.237219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27a44e00-cdb9-466b-9ffe-03e1a0b990f5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:25.290342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd1dfe0-c69e-49d1-bef0-40b45948494f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:25.302169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 835/1667] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:25 2022] POST /identity/v3/users => generated 284 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:25.308992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd1dfe0-c69e-49d1-bef0-40b45948494f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:25.318165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e6e6df-2413-4878-a3f2-be4ad8518bd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:25.319028 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e6e6df-2413-4878-a3f2-be4ad8518bd7 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:25.319492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e6e6df-2413-4878-a3f2-be4ad8518bd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:25.319920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e6e6df-2413-4878-a3f2-be4ad8518bd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:25.321289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e6e6df-2413-4878-a3f2-be4ad8518bd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:25.323419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e6e6df-2413-4878-a3f2-be4ad8518bd7 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:25.339581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a44e00-cdb9-466b-9ffe-03e1a0b990f5 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:25.341070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 833/1668] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:25 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:25.359687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a44e00-cdb9-466b-9ffe-03e1a0b990f5 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:25.374123 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a06781-d969-4285-b9fd-435e4f6f04cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:25.374954 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a06781-d969-4285-b9fd-435e4f6f04cd admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:25.375251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a06781-d969-4285-b9fd-435e4f6f04cd admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:25.375540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a06781-d969-4285-b9fd-435e4f6f04cd admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:25.376418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a06781-d969-4285-b9fd-435e4f6f04cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:25.383184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a06781-d969-4285-b9fd-435e4f6f04cd admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:25.419355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e6e6df-2413-4878-a3f2-be4ad8518bd7 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:25.419791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 836/1669] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:25 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:25.432101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e6e6df-2413-4878-a3f2-be4ad8518bd7 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:25.448958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6848598-6068-4a45-b756-72695a61df15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:25.453171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6848598-6068-4a45-b756-72695a61df15 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:25.453475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6848598-6068-4a45-b756-72695a61df15 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:25.453766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6848598-6068-4a45-b756-72695a61df15 admin admin] PATH_INFO: `/v3/projects/7bca1a9ff6bc494dabcf47b9cab43edd/users/5a40ee55bbf44d2ca0b3333036b173a9/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:25.498428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6848598-6068-4a45-b756-72695a61df15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bca1a9ff6bc494dabcf47b9cab43edd, user_id=5a40ee55bbf44d2ca0b3333036b173a9, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:25.501380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6848598-6068-4a45-b756-72695a61df15 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:25.532309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd559e52-0011-472a-963b-81e8e90be49a None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:25.532713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd559e52-0011-472a-963b-81e8e90be49a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:25.532989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd559e52-0011-472a-963b-81e8e90be49a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:25.535736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 834/1670] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:25 2022] PUT /identity/v3/projects/7bca1a9ff6bc494dabcf47b9cab43edd/users/5a40ee55bbf44d2ca0b3333036b173a9/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:25.580821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-fd559e52-0011-472a-963b-81e8e90be49a None None] MFA Rules not processed for user `5a40ee55bbf44d2ca0b3333036b173a9`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:25.582608 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fd559e52-0011-472a-963b-81e8e90be49a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:25.705556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 837/1671] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:25 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:25.706827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6848598-6068-4a45-b756-72695a61df15 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:25.807135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c5202-0062-4755-aae7-ea75adb6b41b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:25.807453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c5202-0062-4755-aae7-ea75adb6b41b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:25.807731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3c5202-0062-4755-aae7-ea75adb6b41b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:25.820293 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6848598-6068-4a45-b756-72695a61df15 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:25.827174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d571420e-a417-4082-952d-3d5035d08c9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:25.827900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d571420e-a417-4082-952d-3d5035d08c9f admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:25.828246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d571420e-a417-4082-952d-3d5035d08c9f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:25.828509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d571420e-a417-4082-952d-3d5035d08c9f admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:25.829352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d571420e-a417-4082-952d-3d5035d08c9f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:25.832297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d571420e-a417-4082-952d-3d5035d08c9f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:25.870794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-ee3c5202-0062-4755-aae7-ea75adb6b41b None None] MFA Rules not processed for user `34242a2360f14710af2e8dbe6b704ca1`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:25.872552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ee3c5202-0062-4755-aae7-ea75adb6b41b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:25.886568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 835/1672] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:25 2022] POST /identity/v3/projects => generated 365 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:25.887562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d571420e-a417-4082-952d-3d5035d08c9f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:25.936638 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:25.941326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:26.038010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d571420e-a417-4082-952d-3d5035d08c9f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:26.045455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bccff0-a6ec-448c-a1c6-32366bbf57c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:26.046849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bccff0-a6ec-448c-a1c6-32366bbf57c8 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:26.047267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bccff0-a6ec-448c-a1c6-32366bbf57c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:26.047626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bccff0-a6ec-448c-a1c6-32366bbf57c8 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:26.048448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bccff0-a6ec-448c-a1c6-32366bbf57c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:26.051814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bccff0-a6ec-448c-a1c6-32366bbf57c8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:26.069214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 838/1673] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:25 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:26.082372 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c5202-0062-4755-aae7-ea75adb6b41b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:26.087559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ee3c5202-0062-4755-aae7-ea75adb6b41b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:26.121577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 836/1674] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:25 2022] POST /identity/v3/users => generated 289 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:26.124186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ed8fa0-2556-4f86-9d63-b957a01352e8 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:26.126437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ed8fa0-2556-4f86-9d63-b957a01352e8 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:26.128078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ed8fa0-2556-4f86-9d63-b957a01352e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:26.135840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3c5202-0062-4755-aae7-ea75adb6b41b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:26.148265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:26.151632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:26.152038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:26.152427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] PATH_INFO: `/v3/projects/3ba444c7610642f09eb0e5ce86465768` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:26.155968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ba444c7610642f09eb0e5ce86465768)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:26.157851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: WARNING py.warnings [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 29 04:47:26.157851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: warnings.warn(msg) Jul 29 04:47:26.157851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: Jul 29 04:47:26.160702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:26.174480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-52ed8fa0-2556-4f86-9d63-b957a01352e8 None None] MFA Rules not processed for user `34242a2360f14710af2e8dbe6b704ca1`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:26.177813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-52ed8fa0-2556-4f86-9d63-b957a01352e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:26.307047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ba444c7610642f09eb0e5ce86465768', 'request_id': 'req-2ce5ad68-db62-411f-b883-e0c423e1cb99'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:26.324703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ba444c7610642f09eb0e5ce86465768', 'request_id': 'req-2ce5ad68-db62-411f-b883-e0c423e1cb99'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:26.324969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 837/1675] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:26 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:26.326019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ed8fa0-2556-4f86-9d63-b957a01352e8 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:26.339317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ba444c7610642f09eb0e5ce86465768', 'request_id': 'req-2ce5ad68-db62-411f-b883-e0c423e1cb99'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:26.354117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ed8fa0-2556-4f86-9d63-b957a01352e8 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:26.417779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] The token cache is being invalidate because project 3ba444c7610642f09eb0e5ce86465768 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:26.418944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:26.420492 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:26.428137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 839/1676] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:26 2022] DELETE /identity/v3/projects/3ba444c7610642f09eb0e5ce86465768 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:26.428292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:26.431767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:26.495013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bb83bd-b6a7-4f93-b005-929a6f11feaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:26.496406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb83bd-b6a7-4f93-b005-929a6f11feaa admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:26.496802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb83bd-b6a7-4f93-b005-929a6f11feaa admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:26.497157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bb83bd-b6a7-4f93-b005-929a6f11feaa admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:26.498305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bb83bd-b6a7-4f93-b005-929a6f11feaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:26.501222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bb83bd-b6a7-4f93-b005-929a6f11feaa admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:26.525732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 838/1677] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:26 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:26.531280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533cb556-fb44-4fb9-9979-e5a2bf8cdbdb None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:26.531280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533cb556-fb44-4fb9-9979-e5a2bf8cdbdb None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:26.531505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533cb556-fb44-4fb9-9979-e5a2bf8cdbdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:26.567743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-533cb556-fb44-4fb9-9979-e5a2bf8cdbdb None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:26.569792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-533cb556-fb44-4fb9-9979-e5a2bf8cdbdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:26.588641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 839/1678] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:26.589545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533cb556-fb44-4fb9-9979-e5a2bf8cdbdb None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:26.598068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-533cb556-fb44-4fb9-9979-e5a2bf8cdbdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:26.605825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce5ad68-db62-411f-b883-e0c423e1cb99 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:26.618382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba03f62-8fcc-44b5-8c79-fcee328deae7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:26.619885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba03f62-8fcc-44b5-8c79-fcee328deae7 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:26.619885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba03f62-8fcc-44b5-8c79-fcee328deae7 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:26.621310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba03f62-8fcc-44b5-8c79-fcee328deae7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:26.621701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533cb556-fb44-4fb9-9979-e5a2bf8cdbdb None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:26.641788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-dba03f62-8fcc-44b5-8c79-fcee328deae7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:26.655425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70443681-8eaa-474b-81d7-c1371bdf1d59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:26.659182 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70443681-8eaa-474b-81d7-c1371bdf1d59 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:26.659829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70443681-8eaa-474b-81d7-c1371bdf1d59 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:26.660933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70443681-8eaa-474b-81d7-c1371bdf1d59 admin admin] PATH_INFO: `/v3/projects/66c610ea9f794075a7e814fcb0887cfd/users/b1f09bd6fbf84780ad3871e4d83626b2/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:26.711625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70443681-8eaa-474b-81d7-c1371bdf1d59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=66c610ea9f794075a7e814fcb0887cfd, user_id=b1f09bd6fbf84780ad3871e4d83626b2, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:26.718327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70443681-8eaa-474b-81d7-c1371bdf1d59 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:26.751916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 840/1679] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:26 2022] PUT /identity/v3/projects/66c610ea9f794075a7e814fcb0887cfd/users/b1f09bd6fbf84780ad3871e4d83626b2/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:26.753956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ceffd6b-97d4-47f6-b8b9-cbeaf4f9e59d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:26.754549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ceffd6b-97d4-47f6-b8b9-cbeaf4f9e59d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:26.755313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ceffd6b-97d4-47f6-b8b9-cbeaf4f9e59d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:26.804082 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:26.812037 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-1ceffd6b-97d4-47f6-b8b9-cbeaf4f9e59d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:26.814467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-1ceffd6b-97d4-47f6-b8b9-cbeaf4f9e59d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:26.822420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:26.827137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba03f62-8fcc-44b5-8c79-fcee328deae7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:26.828952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba03f62-8fcc-44b5-8c79-fcee328deae7 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:26.853163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 840/1680] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:26 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:26.854021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba03f62-8fcc-44b5-8c79-fcee328deae7 service nova] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:26.872455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba03f62-8fcc-44b5-8c79-fcee328deae7 service nova] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:26.910844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 841/1681] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:26.916578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ceffd6b-97d4-47f6-b8b9-cbeaf4f9e59d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:26.940201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ceffd6b-97d4-47f6-b8b9-cbeaf4f9e59d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:26.954221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2afbcd9-a62b-461e-a257-b4fd1f9e1cea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:26.954883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afbcd9-a62b-461e-a257-b4fd1f9e1cea admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:26.955176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afbcd9-a62b-461e-a257-b4fd1f9e1cea admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:26.955459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2afbcd9-a62b-461e-a257-b4fd1f9e1cea admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:26.956184 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2afbcd9-a62b-461e-a257-b4fd1f9e1cea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:26.956498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2afbcd9-a62b-461e-a257-b4fd1f9e1cea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:26.960208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2afbcd9-a62b-461e-a257-b4fd1f9e1cea admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:26.990812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 842/1682] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:26.995425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2afbcd9-a62b-461e-a257-b4fd1f9e1cea admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:27.009400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d0a3cd-3394-43e8-bb7d-9fdf6fae4ccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:27.009641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d0a3cd-3394-43e8-bb7d-9fdf6fae4ccf admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:27.009764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d0a3cd-3394-43e8-bb7d-9fdf6fae4ccf admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:27.010045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d0a3cd-3394-43e8-bb7d-9fdf6fae4ccf admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:27.010750 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d0a3cd-3394-43e8-bb7d-9fdf6fae4ccf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:27.012363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d0a3cd-3394-43e8-bb7d-9fdf6fae4ccf admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:27.021891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2afbcd9-a62b-461e-a257-b4fd1f9e1cea admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:27.049749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 841/1683] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:26 2022] GET /identity/v3/roles => generated 1420 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:27.050486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0a8b05-6a49-4e6a-becc-f2b153812978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:27.050837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d0a3cd-3394-43e8-bb7d-9fdf6fae4ccf admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:27.053009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0a8b05-6a49-4e6a-becc-f2b153812978 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:27.053345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0a8b05-6a49-4e6a-becc-f2b153812978 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:27.053584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0a8b05-6a49-4e6a-becc-f2b153812978 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:27.054500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0a8b05-6a49-4e6a-becc-f2b153812978 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:27.063972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0a8b05-6a49-4e6a-becc-f2b153812978 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:27.075314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d0a3cd-3394-43e8-bb7d-9fdf6fae4ccf admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:27.102629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04adce26-f5b5-4949-b49c-a4e3cd06a4a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:27.103159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04adce26-f5b5-4949-b49c-a4e3cd06a4a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:27.103498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04adce26-f5b5-4949-b49c-a4e3cd06a4a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:27.104463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04adce26-f5b5-4949-b49c-a4e3cd06a4a6 admin admin] PATH_INFO: `/v3/projects/66c610ea9f794075a7e814fcb0887cfd/users/b1f09bd6fbf84780ad3871e4d83626b2/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:27.108123 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04adce26-f5b5-4949-b49c-a4e3cd06a4a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=66c610ea9f794075a7e814fcb0887cfd, user_id=b1f09bd6fbf84780ad3871e4d83626b2, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:27.111153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04adce26-f5b5-4949-b49c-a4e3cd06a4a6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:27.154759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 842/1684] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:27 2022] PUT /identity/v3/projects/66c610ea9f794075a7e814fcb0887cfd/users/b1f09bd6fbf84780ad3871e4d83626b2/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:27.156712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6193838e-e83a-4aaa-9f6c-64052e373e73 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:27.156712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6193838e-e83a-4aaa-9f6c-64052e373e73 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:27.156942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6193838e-e83a-4aaa-9f6c-64052e373e73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:27.160667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 843/1685] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:26 2022] POST /identity/v3/projects => generated 357 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:27.164539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0a8b05-6a49-4e6a-becc-f2b153812978 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:27.227867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6193838e-e83a-4aaa-9f6c-64052e373e73 None None] MFA Rules not processed for user `b1f09bd6fbf84780ad3871e4d83626b2`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:27.229165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6193838e-e83a-4aaa-9f6c-64052e373e73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:27.291180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0a8b05-6a49-4e6a-becc-f2b153812978 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:27.303101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f83df3-8df0-4eaa-addc-329ca4d89f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:27.305206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f83df3-8df0-4eaa-addc-329ca4d89f94 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:27.305607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f83df3-8df0-4eaa-addc-329ca4d89f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:27.305893 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f83df3-8df0-4eaa-addc-329ca4d89f94 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:27.306709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f83df3-8df0-4eaa-addc-329ca4d89f94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:27.312418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f83df3-8df0-4eaa-addc-329ca4d89f94 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:27.358570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 844/1686] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:27 2022] POST /identity/v3/users => generated 279 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:47:27.362949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f83df3-8df0-4eaa-addc-329ca4d89f94 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:27.394414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f83df3-8df0-4eaa-addc-329ca4d89f94 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:27.397077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 843/1687] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:27 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:27.407185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6193838e-e83a-4aaa-9f6c-64052e373e73 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:27.409475 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6193838e-e83a-4aaa-9f6c-64052e373e73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:27.412355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c99a0a-01cd-4f16-bede-17ff87eabb55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:27.415692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c99a0a-01cd-4f16-bede-17ff87eabb55 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:27.416002 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c99a0a-01cd-4f16-bede-17ff87eabb55 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:27.416286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c99a0a-01cd-4f16-bede-17ff87eabb55 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:27.420876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c99a0a-01cd-4f16-bede-17ff87eabb55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:27.422557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c99a0a-01cd-4f16-bede-17ff87eabb55 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:27.446495 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 845/1688] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:27 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:27.453223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c99a0a-01cd-4f16-bede-17ff87eabb55 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:27.486489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c99a0a-01cd-4f16-bede-17ff87eabb55 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:27.507719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1a6c1f-4a8f-4593-b421-e63b775bc345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:27.508391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1a6c1f-4a8f-4593-b421-e63b775bc345 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:27.508706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1a6c1f-4a8f-4593-b421-e63b775bc345 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:27.508990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1a6c1f-4a8f-4593-b421-e63b775bc345 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:27.509708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1a6c1f-4a8f-4593-b421-e63b775bc345 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:27.511343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1a6c1f-4a8f-4593-b421-e63b775bc345 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:27.558716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6193838e-e83a-4aaa-9f6c-64052e373e73 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:27.573741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e72bef-e691-484d-acec-d9cb13f1e94c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:27.573741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e72bef-e691-484d-acec-d9cb13f1e94c service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:27.573741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e72bef-e691-484d-acec-d9cb13f1e94c service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:27.573741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e72bef-e691-484d-acec-d9cb13f1e94c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:27.574251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 846/1689] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:27 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:27.575238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1a6c1f-4a8f-4593-b421-e63b775bc345 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:27.598903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1a6c1f-4a8f-4593-b421-e63b775bc345 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:27.605965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e72bef-e691-484d-acec-d9cb13f1e94c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:27.607569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e72bef-e691-484d-acec-d9cb13f1e94c service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:27.616399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cacd7e3-3f35-437f-88bb-45550eaa1d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:27.618126 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cacd7e3-3f35-437f-88bb-45550eaa1d29 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:27.619188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cacd7e3-3f35-437f-88bb-45550eaa1d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:27.620477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cacd7e3-3f35-437f-88bb-45550eaa1d29 admin admin] PATH_INFO: `/v3/projects/95e9cb3f60694f9aaffa1b454f5a14e7/users/50203897c78c41528973f0eeaf879d56/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:27.622282 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:27.632826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 844/1690] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:27 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:47:27.664500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cacd7e3-3f35-437f-88bb-45550eaa1d29 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95e9cb3f60694f9aaffa1b454f5a14e7, user_id=50203897c78c41528973f0eeaf879d56, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:27.676375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cacd7e3-3f35-437f-88bb-45550eaa1d29 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:27.723392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 847/1691] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:27 2022] PUT /identity/v3/projects/95e9cb3f60694f9aaffa1b454f5a14e7/users/50203897c78c41528973f0eeaf879d56/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:27.734067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c28500c-8b86-4cc5-b908-6a3cba1793d2 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:27.734583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c28500c-8b86-4cc5-b908-6a3cba1793d2 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:27.735039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c28500c-8b86-4cc5-b908-6a3cba1793d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:27.806820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cacd7e3-3f35-437f-88bb-45550eaa1d29 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:27.813363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-5c28500c-8b86-4cc5-b908-6a3cba1793d2 None None] MFA Rules not processed for user `50203897c78c41528973f0eeaf879d56`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:27.816188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5c28500c-8b86-4cc5-b908-6a3cba1793d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:27.863899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:27.991203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cacd7e3-3f35-437f-88bb-45550eaa1d29 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:28.016627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba007ccb-e59e-4ee2-85a7-cb66f4fc7647 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:28.017480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba007ccb-e59e-4ee2-85a7-cb66f4fc7647 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:28.017875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba007ccb-e59e-4ee2-85a7-cb66f4fc7647 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:28.019375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba007ccb-e59e-4ee2-85a7-cb66f4fc7647 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:28.027079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ba007ccb-e59e-4ee2-85a7-cb66f4fc7647 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:28.044741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 845/1692] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:27 2022] POST /identity/v3/auth/tokens => generated 3224 bytes in 310 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:28.047632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c28500c-8b86-4cc5-b908-6a3cba1793d2 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:28.218293 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba007ccb-e59e-4ee2-85a7-cb66f4fc7647 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:28.221817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba007ccb-e59e-4ee2-85a7-cb66f4fc7647 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:28.243976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c28500c-8b86-4cc5-b908-6a3cba1793d2 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:28.250727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d5f1c1-5861-44a5-8875-b195bb432913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:28.253627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 848/1693] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:27 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 449 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:47:28.263613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d5f1c1-5861-44a5-8875-b195bb432913 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:28.264293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d5f1c1-5861-44a5-8875-b195bb432913 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:28.264762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d5f1c1-5861-44a5-8875-b195bb432913 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:28.265636 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d5f1c1-5861-44a5-8875-b195bb432913 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:28.273414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d5f1c1-5861-44a5-8875-b195bb432913 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:28.346284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 846/1694] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:28 2022] POST /identity/v3/projects => generated 355 bytes in 299 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:28.350867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba007ccb-e59e-4ee2-85a7-cb66f4fc7647 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:28.368714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba007ccb-e59e-4ee2-85a7-cb66f4fc7647 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:28.465535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4164b711-f834-4e70-9848-f9da4d99e5db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:28.466187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4164b711-f834-4e70-9848-f9da4d99e5db admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:28.466617 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4164b711-f834-4e70-9848-f9da4d99e5db admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:28.467177 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4164b711-f834-4e70-9848-f9da4d99e5db admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:28.467779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4164b711-f834-4e70-9848-f9da4d99e5db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:28.470012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4164b711-f834-4e70-9848-f9da4d99e5db admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:28.502808 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 849/1695] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:28 2022] POST /identity/v3/users => generated 284 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:28.505249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d5f1c1-5861-44a5-8875-b195bb432913 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:28.562310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d5f1c1-5861-44a5-8875-b195bb432913 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:28.574239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813b2fbd-2883-4053-8aa6-950830e7dc55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:28.574885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813b2fbd-2883-4053-8aa6-950830e7dc55 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:28.575371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813b2fbd-2883-4053-8aa6-950830e7dc55 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:28.575493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813b2fbd-2883-4053-8aa6-950830e7dc55 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:28.576213 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813b2fbd-2883-4053-8aa6-950830e7dc55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:28.579346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813b2fbd-2883-4053-8aa6-950830e7dc55 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:28.599231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 847/1696] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:28 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:28.600788 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4164b711-f834-4e70-9848-f9da4d99e5db admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:28.623611 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4164b711-f834-4e70-9848-f9da4d99e5db admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:28.639615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d945ef2-0ac0-4a04-b812-9c502b2dc8ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:28.640419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d945ef2-0ac0-4a04-b812-9c502b2dc8ec admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:28.642272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d945ef2-0ac0-4a04-b812-9c502b2dc8ec admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:28.642635 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d945ef2-0ac0-4a04-b812-9c502b2dc8ec admin admin] PATH_INFO: `/v3/projects/8f06f68885a54b40b7db2cb48e93e1d3/users/115d1dea431849b6a1a2fe7c83957d76/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:28.702862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d945ef2-0ac0-4a04-b812-9c502b2dc8ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f06f68885a54b40b7db2cb48e93e1d3, user_id=115d1dea431849b6a1a2fe7c83957d76, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:28.707933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d945ef2-0ac0-4a04-b812-9c502b2dc8ec admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:28.738829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 850/1697] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:28 2022] PUT /identity/v3/projects/8f06f68885a54b40b7db2cb48e93e1d3/users/115d1dea431849b6a1a2fe7c83957d76/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:28.744818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813b2fbd-2883-4053-8aa6-950830e7dc55 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:28.759948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813b2fbd-2883-4053-8aa6-950830e7dc55 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:28.866191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cef57e-d2ab-4ffd-a1df-a800b9740a0b None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:28.866707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cef57e-d2ab-4ffd-a1df-a800b9740a0b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:28.867183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cef57e-d2ab-4ffd-a1df-a800b9740a0b None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:28.869152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 851/1698] 10.222.0.40 () {54 vars in 912 bytes} [Fri Jul 29 04:47:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:47:28.887724 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229caf9e-fab3-4382-9486-d556b64ff32f None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:28.891555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229caf9e-fab3-4382-9486-d556b64ff32f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:28.892053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229caf9e-fab3-4382-9486-d556b64ff32f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:28.927200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744db70a-c922-441d-9112-74228b8d8818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:28.928673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744db70a-c922-441d-9112-74228b8d8818 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:28.928673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744db70a-c922-441d-9112-74228b8d8818 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:28.931234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744db70a-c922-441d-9112-74228b8d8818 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:28.933223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744db70a-c922-441d-9112-74228b8d8818 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:28.937734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744db70a-c922-441d-9112-74228b8d8818 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:28.961275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:28.980509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 848/1699] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:28 2022] GET /identity/v3/roles => generated 1420 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:28.983279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5bafd5-e4e6-4233-9872-121c8ef4e79e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:28.984918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5bafd5-e4e6-4233-9872-121c8ef4e79e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:28.985427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5bafd5-e4e6-4233-9872-121c8ef4e79e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:29.058331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-229caf9e-fab3-4382-9486-d556b64ff32f None None] MFA Rules not processed for user `f6726ef0c4184040a18a76898bb738d1`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:29.058980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-229caf9e-fab3-4382-9486-d556b64ff32f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:29.101780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-fc5bafd5-e4e6-4233-9872-121c8ef4e79e None None] MFA Rules not processed for user `5f9d45265c6f47609f9f8abcf3526ff0`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:29.113868 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fc5bafd5-e4e6-4233-9872-121c8ef4e79e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:29.129604 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:29.240588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 852/1700] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:47:28 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:29.241842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229caf9e-fab3-4382-9486-d556b64ff32f None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:29.278504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229caf9e-fab3-4382-9486-d556b64ff32f None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:29.285527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cbf616-c4d3-4030-8784-f53c7d8dd79d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:29.286598 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cbf616-c4d3-4030-8784-f53c7d8dd79d admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:29.287245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cbf616-c4d3-4030-8784-f53c7d8dd79d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:29.287990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cbf616-c4d3-4030-8784-f53c7d8dd79d admin admin] PATH_INFO: `/v3/projects/8f06f68885a54b40b7db2cb48e93e1d3/users/115d1dea431849b6a1a2fe7c83957d76/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:29.294664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cbf616-c4d3-4030-8784-f53c7d8dd79d admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f06f68885a54b40b7db2cb48e93e1d3, user_id=115d1dea431849b6a1a2fe7c83957d76, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:29.302088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cbf616-c4d3-4030-8784-f53c7d8dd79d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:29.378854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 853/1701] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:29 2022] PUT /identity/v3/projects/8f06f68885a54b40b7db2cb48e93e1d3/users/115d1dea431849b6a1a2fe7c83957d76/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:29.384463 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e039ab7-ded8-4412-80d8-5fa031f0becf None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:29.385296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e039ab7-ded8-4412-80d8-5fa031f0becf None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:29.385811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e039ab7-ded8-4412-80d8-5fa031f0becf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:29.424845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 849/1702] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:28 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 444 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:29.436054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6480a8c5-c865-4386-b8f5-97762667bff7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:29.436357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6480a8c5-c865-4386-b8f5-97762667bff7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:29.436661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6480a8c5-c865-4386-b8f5-97762667bff7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:29.482666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-6e039ab7-ded8-4412-80d8-5fa031f0becf None None] MFA Rules not processed for user `115d1dea431849b6a1a2fe7c83957d76`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:29.487915 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-6e039ab7-ded8-4412-80d8-5fa031f0becf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:29.509918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6480a8c5-c865-4386-b8f5-97762667bff7 None None] MFA Rules not processed for user `5f9d45265c6f47609f9f8abcf3526ff0`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:29.511221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6480a8c5-c865-4386-b8f5-97762667bff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:29.661350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 850/1703] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:29.662297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6480a8c5-c865-4386-b8f5-97762667bff7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:29.675062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6480a8c5-c865-4386-b8f5-97762667bff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:29.742068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 854/1704] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:29 2022] POST /identity/v3/auth/tokens => generated 3289 bytes in 358 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:29.742068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e039ab7-ded8-4412-80d8-5fa031f0becf None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:29.743558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-6e039ab7-ded8-4412-80d8-5fa031f0becf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:29.856630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6480a8c5-c865-4386-b8f5-97762667bff7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:29.860052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3036736c-eedb-4937-b5b1-8cb72bced4cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:29.860857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3036736c-eedb-4937-b5b1-8cb72bced4cc service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:29.861255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3036736c-eedb-4937-b5b1-8cb72bced4cc service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:29.861653 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3036736c-eedb-4937-b5b1-8cb72bced4cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:29.886379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3036736c-eedb-4937-b5b1-8cb72bced4cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:29.889848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3036736c-eedb-4937-b5b1-8cb72bced4cc service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:29.926214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 851/1705] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:29 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:29.926214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3036736c-eedb-4937-b5b1-8cb72bced4cc service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:29.975961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e039ab7-ded8-4412-80d8-5fa031f0becf None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:29.993391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b99a78-0379-47e3-b3b8-8771b66d9942 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:29.994405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b99a78-0379-47e3-b3b8-8771b66d9942 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:29.995143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b99a78-0379-47e3-b3b8-8771b66d9942 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:29.997389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b99a78-0379-47e3-b3b8-8771b66d9942 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:30.018394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b99a78-0379-47e3-b3b8-8771b66d9942 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:30.020148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b99a78-0379-47e3-b3b8-8771b66d9942 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:30.042266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:30.046693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 855/1706] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:29 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:30.048044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bfa6c1-80da-4514-8176-e80db0f6d216 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:30.053158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bfa6c1-80da-4514-8176-e80db0f6d216 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:30.053487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bfa6c1-80da-4514-8176-e80db0f6d216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:30.091082 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3036736c-eedb-4937-b5b1-8cb72bced4cc service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:30.106301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c40f37c-96f5-47d5-b8ee-8d8458b1852c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:30.107118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c40f37c-96f5-47d5-b8ee-8d8458b1852c service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:30.107517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c40f37c-96f5-47d5-b8ee-8d8458b1852c service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:30.107912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c40f37c-96f5-47d5-b8ee-8d8458b1852c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:30.140494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c40f37c-96f5-47d5-b8ee-8d8458b1852c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:30.142093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c40f37c-96f5-47d5-b8ee-8d8458b1852c service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:30.154027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f7bfa6c1-80da-4514-8176-e80db0f6d216 None None] MFA Rules not processed for user `8b7751b93f88424abebe6859f46f8351`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:30.157469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f7bfa6c1-80da-4514-8176-e80db0f6d216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:30.187054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 852/1707] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:29 2022] GET /identity/v3/auth/tokens => generated 3289 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:30.227959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:30.341982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 856/1708] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:30 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:30.358370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c40f37c-96f5-47d5-b8ee-8d8458b1852c service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:30.374547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c40f37c-96f5-47d5-b8ee-8d8458b1852c service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:30.399564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322991ca-6a78-496d-a3ff-f8a1d088126d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:30.400289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322991ca-6a78-496d-a3ff-f8a1d088126d service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:30.401721 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322991ca-6a78-496d-a3ff-f8a1d088126d service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:30.402081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322991ca-6a78-496d-a3ff-f8a1d088126d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:30.575963 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322991ca-6a78-496d-a3ff-f8a1d088126d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:30.582044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322991ca-6a78-496d-a3ff-f8a1d088126d service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:30.622978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 853/1709] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:30 2022] GET /identity/v3/auth/tokens => generated 3224 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:31.110612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646ec5f8-5951-4690-9493-8f10526f9c14 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:31.111098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646ec5f8-5951-4690-9493-8f10526f9c14 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:31.111181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646ec5f8-5951-4690-9493-8f10526f9c14 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:31.113224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 857/1710] 10.222.0.40 () {54 vars in 912 bytes} [Fri Jul 29 04:47:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:47:31.131747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f55604f-5467-4899-a3b0-495a4da3a7c5 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:31.132192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f55604f-5467-4899-a3b0-495a4da3a7c5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:31.132261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f55604f-5467-4899-a3b0-495a4da3a7c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:31.181668 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6f55604f-5467-4899-a3b0-495a4da3a7c5 None None] MFA Rules not processed for user `f6726ef0c4184040a18a76898bb738d1`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:31.182918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6f55604f-5467-4899-a3b0-495a4da3a7c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:31.220282 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 854/1711] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:47:31 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:31.232057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646ec5f8-5951-4690-9493-8f10526f9c14 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:31.234179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-646ec5f8-5951-4690-9493-8f10526f9c14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:31.265583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646ec5f8-5951-4690-9493-8f10526f9c14 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:31.285453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eab4f5a-6418-4cd9-8ab3-aedcab11d175 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:31.286229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eab4f5a-6418-4cd9-8ab3-aedcab11d175 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:31.287111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eab4f5a-6418-4cd9-8ab3-aedcab11d175 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:31.287475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eab4f5a-6418-4cd9-8ab3-aedcab11d175 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:31.306343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eab4f5a-6418-4cd9-8ab3-aedcab11d175 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:31.318764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eab4f5a-6418-4cd9-8ab3-aedcab11d175 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:31.333309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 858/1712] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:31 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:31.820866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e735f225-b9b8-4205-b6d6-2058cec8fa8d None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:31.821535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e735f225-b9b8-4205-b6d6-2058cec8fa8d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:31.822104 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e735f225-b9b8-4205-b6d6-2058cec8fa8d None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:31.823946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 855/1713] 10.222.0.40 () {54 vars in 961 bytes} [Fri Jul 29 04:47:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:47:31.828331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eab4f5a-6418-4cd9-8ab3-aedcab11d175 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:31.832941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5c42ca-bde4-4f4b-9fa4-0bae762ec45f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:31.834408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5c42ca-bde4-4f4b-9fa4-0bae762ec45f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:31.835068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5c42ca-bde4-4f4b-9fa4-0bae762ec45f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:31.858748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eab4f5a-6418-4cd9-8ab3-aedcab11d175 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:31.863008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-fb5c42ca-bde4-4f4b-9fa4-0bae762ec45f None None] MFA Rules not processed for user `e3da5e428049419a9b35494655f52484`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:31.864308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fb5c42ca-bde4-4f4b-9fa4-0bae762ec45f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:31.875519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602944ed-e386-4eee-a358-c257cc3a591a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:31.876175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602944ed-e386-4eee-a358-c257cc3a591a service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:31.876466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602944ed-e386-4eee-a358-c257cc3a591a service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:31.876788 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602944ed-e386-4eee-a358-c257cc3a591a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:31.900517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602944ed-e386-4eee-a358-c257cc3a591a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:31.902210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602944ed-e386-4eee-a358-c257cc3a591a service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:31.924172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 859/1714] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:31 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:31.984094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 856/1715] 10.222.0.40 () {58 vars in 1058 bytes} [Fri Jul 29 04:47:31 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:31.995920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602944ed-e386-4eee-a358-c257cc3a591a service nova] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:32.022010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602944ed-e386-4eee-a358-c257cc3a591a service nova] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:32.036079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d7b0cc-7939-40a2-8f50-e1fc91b39ea4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:32.038587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d7b0cc-7939-40a2-8f50-e1fc91b39ea4 service placement] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:32.038939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d7b0cc-7939-40a2-8f50-e1fc91b39ea4 service placement] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:32.039817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d7b0cc-7939-40a2-8f50-e1fc91b39ea4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:32.044841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a3d7b0cc-7939-40a2-8f50-e1fc91b39ea4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:32.081492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d7b0cc-7939-40a2-8f50-e1fc91b39ea4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:32.084554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d7b0cc-7939-40a2-8f50-e1fc91b39ea4 service placement] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:32.109707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 860/1716] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:31 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:32.813602 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e099c6c0-3841-4004-8170-193603a6e6da None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:32.814104 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e099c6c0-3841-4004-8170-193603a6e6da None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:32.814297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e099c6c0-3841-4004-8170-193603a6e6da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:32.880783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e099c6c0-3841-4004-8170-193603a6e6da None None] MFA Rules not processed for user `5f9d45265c6f47609f9f8abcf3526ff0`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:32.886991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e099c6c0-3841-4004-8170-193603a6e6da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:32.926176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 857/1717] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:32 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:35.623694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125e1996-d1f5-46bd-9e0f-6f3d50ea8ccc None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:35.624194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125e1996-d1f5-46bd-9e0f-6f3d50ea8ccc None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:35.624282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125e1996-d1f5-46bd-9e0f-6f3d50ea8ccc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:35.674683 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-125e1996-d1f5-46bd-9e0f-6f3d50ea8ccc None None] MFA Rules not processed for user `572408fd157844dcae13000d8e9c4b7a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:35.676049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-125e1996-d1f5-46bd-9e0f-6f3d50ea8ccc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:35.875748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 861/1718] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:35 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:35.892538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e099c6c0-3841-4004-8170-193603a6e6da None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:35.919570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e099c6c0-3841-4004-8170-193603a6e6da None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:35.969467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a987e762-c63b-4e25-9a66-a2c1b7b1d7f9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:35.971737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a987e762-c63b-4e25-9a66-a2c1b7b1d7f9 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:35.972301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a987e762-c63b-4e25-9a66-a2c1b7b1d7f9 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:35.973639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a987e762-c63b-4e25-9a66-a2c1b7b1d7f9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:36.005368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a987e762-c63b-4e25-9a66-a2c1b7b1d7f9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:36.013011 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a987e762-c63b-4e25-9a66-a2c1b7b1d7f9 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:36.025830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 858/1719] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:35 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:36.197240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d623a3f6-4255-4ab3-b079-7973a10fe452 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:36.197554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d623a3f6-4255-4ab3-b079-7973a10fe452 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:36.197855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d623a3f6-4255-4ab3-b079-7973a10fe452 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:36.241599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d623a3f6-4255-4ab3-b079-7973a10fe452 None None] MFA Rules not processed for user `8b7751b93f88424abebe6859f46f8351`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:36.243119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d623a3f6-4255-4ab3-b079-7973a10fe452 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:36.254533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 862/1720] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:36 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:38.726335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4198ed71-fb70-4953-b698-2518cbc3f173 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:38.727407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4198ed71-fb70-4953-b698-2518cbc3f173 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:38.727843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4198ed71-fb70-4953-b698-2518cbc3f173 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:38.946554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4198ed71-fb70-4953-b698-2518cbc3f173 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:38.948951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4198ed71-fb70-4953-b698-2518cbc3f173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:39.022035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:39.242765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 859/1721] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:47:38 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 517 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:39.498220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d623a3f6-4255-4ab3-b079-7973a10fe452 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:39.499821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d623a3f6-4255-4ab3-b079-7973a10fe452 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:39.529835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4198ed71-fb70-4953-b698-2518cbc3f173 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:39.533694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4198ed71-fb70-4953-b698-2518cbc3f173 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:39.773497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d623a3f6-4255-4ab3-b079-7973a10fe452 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:39.799645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:39.800526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:39.800659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:39.805853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] PATH_INFO: `/v3/users/865e73a677c0412ab5619e92905526dd` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:39.807887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=865e73a677c0412ab5619e92905526dd)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:39.810095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:39.842725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:39.907277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4198ed71-fb70-4953-b698-2518cbc3f173 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:39.920907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cef9966-adaa-4198-97f7-aafcba3f69ea service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:39.921963 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cef9966-adaa-4198-97f7-aafcba3f69ea service swift] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:39.922286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cef9966-adaa-4198-97f7-aafcba3f69ea service swift] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:39.922574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cef9966-adaa-4198-97f7-aafcba3f69ea service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:39.950669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cef9966-adaa-4198-97f7-aafcba3f69ea service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:39.952377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cef9966-adaa-4198-97f7-aafcba3f69ea service swift] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:39.974627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '865e73a677c0412ab5619e92905526dd'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:39.976472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '865e73a677c0412ab5619e92905526dd'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:39.977261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '865e73a677c0412ab5619e92905526dd'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:39.980849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 860/1722] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:47:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:47:40.027202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '865e73a677c0412ab5619e92905526dd'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:40.049692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '865e73a677c0412ab5619e92905526dd'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:40.093526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 863/1723] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:39 2022] DELETE /identity/v3/users/865e73a677c0412ab5619e92905526dd => generated 0 bytes in 593 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:40.095210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5f56f-184d-4316-aeaa-0532bfeee584 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:40.095465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5f56f-184d-4316-aeaa-0532bfeee584 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:40.095779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5f56f-184d-4316-aeaa-0532bfeee584 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:40.189606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-abe5f56f-184d-4316-aeaa-0532bfeee584 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:40.189606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-abe5f56f-184d-4316-aeaa-0532bfeee584 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:40.288896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:40.290251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:40.454908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 861/1724] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 361 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:40.570431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf4ec68-48ca-4ec5-a8f9-14d645c271c2 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:40.584136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:40.588457 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:40.589360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:40.590480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] PATH_INFO: `/v3/users/5a40ee55bbf44d2ca0b3333036b173a9` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:40.594307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a40ee55bbf44d2ca0b3333036b173a9)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:40.601294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:40.742049 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe5f56f-184d-4316-aeaa-0532bfeee584 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:40.753339 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-abe5f56f-184d-4316-aeaa-0532bfeee584 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:40.893556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a40ee55bbf44d2ca0b3333036b173a9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:40.899577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a40ee55bbf44d2ca0b3333036b173a9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:40.901001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a40ee55bbf44d2ca0b3333036b173a9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:40.964586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a40ee55bbf44d2ca0b3333036b173a9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:41.001140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:41.024399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e888c06c-e53b-41c0-8a60-663ae5d17c57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a40ee55bbf44d2ca0b3333036b173a9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:41.096943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 864/1725] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:40 2022] DELETE /identity/v3/users/5a40ee55bbf44d2ca0b3333036b173a9 => generated 0 bytes in 806 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:47:41.099810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4251cda-6c70-4e17-b027-9bbb36cc7ea7 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:41.101906 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4251cda-6c70-4e17-b027-9bbb36cc7ea7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:41.103456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4251cda-6c70-4e17-b027-9bbb36cc7ea7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:41.127022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:41.139761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe5f56f-184d-4316-aeaa-0532bfeee584 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:41.155628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a4251cda-6c70-4e17-b027-9bbb36cc7ea7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:41.158196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a4251cda-6c70-4e17-b027-9bbb36cc7ea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:41.300492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 865/1726] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:41.352054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f5cd52-71b9-4311-99c0-c0534d317a4d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:41.353035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f5cd52-71b9-4311-99c0-c0534d317a4d service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:41.353346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f5cd52-71b9-4311-99c0-c0534d317a4d service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:41.353625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f5cd52-71b9-4311-99c0-c0534d317a4d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:41.356967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f5f5cd52-71b9-4311-99c0-c0534d317a4d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:41.385199 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f5cd52-71b9-4311-99c0-c0534d317a4d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:41.387703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f5cd52-71b9-4311-99c0-c0534d317a4d service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:41.406191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 862/1727] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 670 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:41.439154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4251cda-6c70-4e17-b027-9bbb36cc7ea7 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:41.441757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a4251cda-6c70-4e17-b027-9bbb36cc7ea7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:41.467244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4251cda-6c70-4e17-b027-9bbb36cc7ea7 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:41.477430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1359df16-3b51-49b8-a989-ea261e2b936a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:41.479421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1359df16-3b51-49b8-a989-ea261e2b936a service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:41.481488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1359df16-3b51-49b8-a989-ea261e2b936a service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:41.481790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1359df16-3b51-49b8-a989-ea261e2b936a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:41.529990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1359df16-3b51-49b8-a989-ea261e2b936a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:41.545388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1359df16-3b51-49b8-a989-ea261e2b936a service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:41.577978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef8a945-89af-4947-923b-da6525527c0b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:41.578270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef8a945-89af-4947-923b-da6525527c0b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:41.578554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef8a945-89af-4947-923b-da6525527c0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:41.589422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 866/1728] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:41.675826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-5ef8a945-89af-4947-923b-da6525527c0b None None] MFA Rules not processed for user `572408fd157844dcae13000d8e9c4b7a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:41.680539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5ef8a945-89af-4947-923b-da6525527c0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:42.015875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 863/1729] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 439 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:42.472698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d06d2c-92cf-4c1d-bb3f-85d1e4d33be8 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:42.473046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d06d2c-92cf-4c1d-bb3f-85d1e4d33be8 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:42.473344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d06d2c-92cf-4c1d-bb3f-85d1e4d33be8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:42.514584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d8d06d2c-92cf-4c1d-bb3f-85d1e4d33be8 None None] MFA Rules not processed for user `f6726ef0c4184040a18a76898bb738d1`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:42.515875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d8d06d2c-92cf-4c1d-bb3f-85d1e4d33be8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:42.563167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 867/1730] 10.222.0.64 () {58 vars in 1015 bytes} [Fri Jul 29 04:47:42 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:42.809979 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef8a945-89af-4947-923b-da6525527c0b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:42.811421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5ef8a945-89af-4947-923b-da6525527c0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:42.862900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef8a945-89af-4947-923b-da6525527c0b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:42.870865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39081031-e3a0-4f64-8837-315732413982 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:42.871520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39081031-e3a0-4f64-8837-315732413982 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:42.871807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39081031-e3a0-4f64-8837-315732413982 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:42.872090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39081031-e3a0-4f64-8837-315732413982 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:42.903969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39081031-e3a0-4f64-8837-315732413982 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:42.907617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39081031-e3a0-4f64-8837-315732413982 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:42.947908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 864/1731] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:42 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:45.413202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d06d2c-92cf-4c1d-bb3f-85d1e4d33be8 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:45.415058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d8d06d2c-92cf-4c1d-bb3f-85d1e4d33be8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:45.450095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d06d2c-92cf-4c1d-bb3f-85d1e4d33be8 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:45.468819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:45.470658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:45.472523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:45.476295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] PATH_INFO: `/v3/projects/7bca1a9ff6bc494dabcf47b9cab43edd` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:45.480524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bca1a9ff6bc494dabcf47b9cab43edd)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:45.485318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:45.628851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bca1a9ff6bc494dabcf47b9cab43edd', 'request_id': 'req-4a5e2c0e-7a60-411b-8673-239580850cff'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:45.674835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bca1a9ff6bc494dabcf47b9cab43edd', 'request_id': 'req-4a5e2c0e-7a60-411b-8673-239580850cff'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:45.698240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bca1a9ff6bc494dabcf47b9cab43edd', 'request_id': 'req-4a5e2c0e-7a60-411b-8673-239580850cff'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:45.698471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39081031-e3a0-4f64-8837-315732413982 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:45.701961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-39081031-e3a0-4f64-8837-315732413982 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:45.742567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39081031-e3a0-4f64-8837-315732413982 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:45.763165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] The token cache is being invalidate because project 7bca1a9ff6bc494dabcf47b9cab43edd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:45.763525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:45.772478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:45.776965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 868/1732] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:45 2022] DELETE /identity/v3/projects/7bca1a9ff6bc494dabcf47b9cab43edd => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:45.782550 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:45.786401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:45.951750 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:45.954098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:45.954972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:45.955694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] PATH_INFO: `/v3/projects/84bbb598d5204cad9c8cb378bc09f1c3` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:45.962802 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] RBAC: Authorizing `identity:delete_project(project_id=84bbb598d5204cad9c8cb378bc09f1c3)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:45.971356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:46.051328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5e2c0e-7a60-411b-8673-239580850cff admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:46.066508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:46.068437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:46.069048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:46.070159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] PATH_INFO: `/v3/users/b1f09bd6fbf84780ad3871e4d83626b2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:46.073542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1f09bd6fbf84780ad3871e4d83626b2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:46.080631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:46.091240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:46.127280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:46.216801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84bbb598d5204cad9c8cb378bc09f1c3', 'request_id': 'req-c9ca5b84-b308-4e1c-a703-defef533aaae'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:46.319368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84bbb598d5204cad9c8cb378bc09f1c3', 'request_id': 'req-c9ca5b84-b308-4e1c-a703-defef533aaae'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:46.388211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84bbb598d5204cad9c8cb378bc09f1c3', 'request_id': 'req-c9ca5b84-b308-4e1c-a703-defef533aaae'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:46.454391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1f09bd6fbf84780ad3871e4d83626b2'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:46.462326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1f09bd6fbf84780ad3871e4d83626b2'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:46.464937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1f09bd6fbf84780ad3871e4d83626b2'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:46.476319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] The token cache is being invalidate because project 84bbb598d5204cad9c8cb378bc09f1c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:46.485911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:46.489490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c9ca5b84-b308-4e1c-a703-defef533aaae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:46.502601 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 865/1733] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:45 2022] DELETE /identity/v3/projects/84bbb598d5204cad9c8cb378bc09f1c3 => generated 0 bytes in 805 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:47:46.513941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3b32f8-9056-4d7d-9647-cb9eef99ad83 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:46.514234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3b32f8-9056-4d7d-9647-cb9eef99ad83 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:46.514514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3b32f8-9056-4d7d-9647-cb9eef99ad83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:46.554959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1f09bd6fbf84780ad3871e4d83626b2'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:46.598752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8c568cad-ba6d-43c5-a833-055b4693b5a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1f09bd6fbf84780ad3871e4d83626b2'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:46.634026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0d3b32f8-9056-4d7d-9647-cb9eef99ad83 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:46.640650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0d3b32f8-9056-4d7d-9647-cb9eef99ad83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:46.647889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 869/1734] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:45 2022] DELETE /identity/v3/users/b1f09bd6fbf84780ad3871e4d83626b2 => generated 0 bytes in 866 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:46.891695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 866/1735] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 384 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:46.900570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9251191e-1d61-4e7b-8628-a47902f5ddb1 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:46.901209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9251191e-1d61-4e7b-8628-a47902f5ddb1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:46.901663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9251191e-1d61-4e7b-8628-a47902f5ddb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:46.956595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9251191e-1d61-4e7b-8628-a47902f5ddb1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:46.958428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9251191e-1d61-4e7b-8628-a47902f5ddb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:46.984870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 870/1736] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:46.990310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3b32f8-9056-4d7d-9647-cb9eef99ad83 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:47.008268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3b32f8-9056-4d7d-9647-cb9eef99ad83 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:47.017528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14bfdcc-503a-4b11-967a-9b6bde97fdb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:47.018111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14bfdcc-503a-4b11-967a-9b6bde97fdb5 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:47.022660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14bfdcc-503a-4b11-967a-9b6bde97fdb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:47.023001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14bfdcc-503a-4b11-967a-9b6bde97fdb5 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:47.023774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14bfdcc-503a-4b11-967a-9b6bde97fdb5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:47.024097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14bfdcc-503a-4b11-967a-9b6bde97fdb5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:47.033976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14bfdcc-503a-4b11-967a-9b6bde97fdb5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:47.062217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 867/1737] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:47.063796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9251191e-1d61-4e7b-8628-a47902f5ddb1 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:47.079931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9251191e-1d61-4e7b-8628-a47902f5ddb1 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:47.104618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58466c34-01aa-45fa-b84b-a0063715c82b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:47.105326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58466c34-01aa-45fa-b84b-a0063715c82b admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:47.105594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58466c34-01aa-45fa-b84b-a0063715c82b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:47.105876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58466c34-01aa-45fa-b84b-a0063715c82b admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:47.106555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58466c34-01aa-45fa-b84b-a0063715c82b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:47.108496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58466c34-01aa-45fa-b84b-a0063715c82b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:47.210754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 871/1738] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:47 2022] POST /identity/v3/projects => generated 403 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:47.214650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14bfdcc-503a-4b11-967a-9b6bde97fdb5 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:47.235441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14bfdcc-503a-4b11-967a-9b6bde97fdb5 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:47.389658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ae97b8-7dad-4201-a232-835757b63f7c None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:47.390034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ae97b8-7dad-4201-a232-835757b63f7c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:47.390388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ae97b8-7dad-4201-a232-835757b63f7c None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:47.391830 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 872/1739] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:47:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:47:47.451542 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad11dcb3-525f-4da5-a9de-8075a2c54d86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:47.452194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad11dcb3-525f-4da5-a9de-8075a2c54d86 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:47.452481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad11dcb3-525f-4da5-a9de-8075a2c54d86 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:47.452796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad11dcb3-525f-4da5-a9de-8075a2c54d86 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:47.453495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad11dcb3-525f-4da5-a9de-8075a2c54d86 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:47.469683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad11dcb3-525f-4da5-a9de-8075a2c54d86 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:47.562285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 868/1740] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:47 2022] POST /identity/v3/users => generated 302 bytes in 348 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:47.565484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ae97b8-7dad-4201-a232-835757b63f7c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:47.578622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fca5bfe-ee53-4fab-8bf6-4d2edffcabc2 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:47.578920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fca5bfe-ee53-4fab-8bf6-4d2edffcabc2 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:47.579253 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fca5bfe-ee53-4fab-8bf6-4d2edffcabc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:47.615771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ae97b8-7dad-4201-a232-835757b63f7c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:47.636111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7507dc7-01f0-446d-b8a8-1e5d85c89d4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:47.640103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7507dc7-01f0-446d-b8a8-1e5d85c89d4f admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:47.642490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7507dc7-01f0-446d-b8a8-1e5d85c89d4f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:47.642786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7507dc7-01f0-446d-b8a8-1e5d85c89d4f admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:47.643540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7507dc7-01f0-446d-b8a8-1e5d85c89d4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:47.656508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7507dc7-01f0-446d-b8a8-1e5d85c89d4f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:47.710137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-9fca5bfe-ee53-4fab-8bf6-4d2edffcabc2 None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:47.711959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9fca5bfe-ee53-4fab-8bf6-4d2edffcabc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:47.721510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 873/1741] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:47 2022] GET /identity/v3/roles => generated 1420 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:47.732360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7507dc7-01f0-446d-b8a8-1e5d85c89d4f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:47.784383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7507dc7-01f0-446d-b8a8-1e5d85c89d4f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:47.809523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de945b62-3aae-42c7-944b-ac42d05697c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:47.810179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de945b62-3aae-42c7-944b-ac42d05697c4 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:47.812631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de945b62-3aae-42c7-944b-ac42d05697c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:47.816004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de945b62-3aae-42c7-944b-ac42d05697c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:47.835062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de945b62-3aae-42c7-944b-ac42d05697c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:47.868742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de945b62-3aae-42c7-944b-ac42d05697c4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:47.887958 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:47.914206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 874/1742] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:47 2022] GET /identity/v3/roles => generated 1420 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:47.919709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de945b62-3aae-42c7-944b-ac42d05697c4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:47.960841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de945b62-3aae-42c7-944b-ac42d05697c4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:47.987881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a95683-d9e2-4daa-ad33-33304fa36f6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:47.991474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a95683-d9e2-4daa-ad33-33304fa36f6d admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:47.992955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a95683-d9e2-4daa-ad33-33304fa36f6d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:47.997999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a95683-d9e2-4daa-ad33-33304fa36f6d admin admin] PATH_INFO: `/v3/projects/f51e2ccad2b749a4bc199e678510b53f/users/3082a1cb1c0a442ea476d6a8419a1846/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:48.065109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a95683-d9e2-4daa-ad33-33304fa36f6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f51e2ccad2b749a4bc199e678510b53f, user_id=3082a1cb1c0a442ea476d6a8419a1846, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:48.068971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a95683-d9e2-4daa-ad33-33304fa36f6d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:48.077483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 869/1743] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:47:47 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 500 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:48.110513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 875/1744] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:47 2022] PUT /identity/v3/projects/f51e2ccad2b749a4bc199e678510b53f/users/3082a1cb1c0a442ea476d6a8419a1846/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:48.116220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d38302-d5b5-4d7d-870c-2e284f877b92 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:48.116812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d38302-d5b5-4d7d-870c-2e284f877b92 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:48.117292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d38302-d5b5-4d7d-870c-2e284f877b92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:48.152539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-60d38302-d5b5-4d7d-870c-2e284f877b92 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:48.158210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-60d38302-d5b5-4d7d-870c-2e284f877b92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:48.311268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 870/1745] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:48.358436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a95683-d9e2-4daa-ad33-33304fa36f6d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:48.360787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-46a95683-d9e2-4daa-ad33-33304fa36f6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:48.592203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a95683-d9e2-4daa-ad33-33304fa36f6d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:48.601859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09bf5ba-333c-4295-992e-041d3bd2daef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:48.602760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09bf5ba-333c-4295-992e-041d3bd2daef service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:48.603245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09bf5ba-333c-4295-992e-041d3bd2daef service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:48.607914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09bf5ba-333c-4295-992e-041d3bd2daef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:48.623942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09bf5ba-333c-4295-992e-041d3bd2daef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:48.626639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09bf5ba-333c-4295-992e-041d3bd2daef service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:48.641719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 876/1746] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 283 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:50.063503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d38302-d5b5-4d7d-870c-2e284f877b92 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:50.065605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-60d38302-d5b5-4d7d-870c-2e284f877b92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:50.183294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d38302-d5b5-4d7d-870c-2e284f877b92 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:50.190948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df928d0-e519-462c-8953-e2697675c976 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:50.191173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df928d0-e519-462c-8953-e2697675c976 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:50.191320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df928d0-e519-462c-8953-e2697675c976 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:50.193294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df928d0-e519-462c-8953-e2697675c976 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:50.205350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df928d0-e519-462c-8953-e2697675c976 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:50.208321 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df928d0-e519-462c-8953-e2697675c976 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:50.249949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 871/1747] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:50 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:51.399288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09bf5ba-333c-4295-992e-041d3bd2daef service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:51.400823 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a09bf5ba-333c-4295-992e-041d3bd2daef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:51.427127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09bf5ba-333c-4295-992e-041d3bd2daef service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:51.447176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:51.448224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:51.448799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:51.449305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] PATH_INFO: `/v3/projects/66c610ea9f794075a7e814fcb0887cfd` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:51.457968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] RBAC: Authorizing `identity:delete_project(project_id=66c610ea9f794075a7e814fcb0887cfd)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:51.461648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:51.497687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:DELETE=10,keystone:SELECT=87,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:51.584870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66c610ea9f794075a7e814fcb0887cfd', 'request_id': 'req-78f124bb-5e40-4e86-8de8-a85ed238baaa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:51.607585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66c610ea9f794075a7e814fcb0887cfd', 'request_id': 'req-78f124bb-5e40-4e86-8de8-a85ed238baaa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:51.627093 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66c610ea9f794075a7e814fcb0887cfd', 'request_id': 'req-78f124bb-5e40-4e86-8de8-a85ed238baaa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:51.674172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] The token cache is being invalidate because project 66c610ea9f794075a7e814fcb0887cfd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:51.674678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:51.677642 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-78f124bb-5e40-4e86-8de8-a85ed238baaa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:51.680995 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 877/1748] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:51 2022] DELETE /identity/v3/projects/66c610ea9f794075a7e814fcb0887cfd => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:51.687486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70432586-b7fd-4590-8b78-b6dd52f3a4d7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:51.691044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70432586-b7fd-4590-8b78-b6dd52f3a4d7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:51.691520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70432586-b7fd-4590-8b78-b6dd52f3a4d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:51.726123 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-70432586-b7fd-4590-8b78-b6dd52f3a4d7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:51.727669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-70432586-b7fd-4590-8b78-b6dd52f3a4d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:51.827177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:51.877095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 872/1749] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:51.885404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c378ef7f-f1fc-44fc-a8e8-d912c39b3c25 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:51.886189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c378ef7f-f1fc-44fc-a8e8-d912c39b3c25 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:51.886676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c378ef7f-f1fc-44fc-a8e8-d912c39b3c25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:51.935179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c378ef7f-f1fc-44fc-a8e8-d912c39b3c25 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:51.937032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c378ef7f-f1fc-44fc-a8e8-d912c39b3c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:51.954164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 878/1750] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:51.961952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70432586-b7fd-4590-8b78-b6dd52f3a4d7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:51.983320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70432586-b7fd-4590-8b78-b6dd52f3a4d7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:51.989401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c76b2c-1b80-450a-8c61-f60dd5121bab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:51.990986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c76b2c-1b80-450a-8c61-f60dd5121bab admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:51.991530 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c76b2c-1b80-450a-8c61-f60dd5121bab admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:51.992188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c76b2c-1b80-450a-8c61-f60dd5121bab admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:51.993199 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c76b2c-1b80-450a-8c61-f60dd5121bab admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:52.001289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c76b2c-1b80-450a-8c61-f60dd5121bab admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:52.006033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c76b2c-1b80-450a-8c61-f60dd5121bab admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:52.039269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 873/1751] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:52.039537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c378ef7f-f1fc-44fc-a8e8-d912c39b3c25 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:52.055102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c378ef7f-f1fc-44fc-a8e8-d912c39b3c25 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:52.065238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b59709e-7dee-4201-bf56-33cdaf687f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:52.066070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59709e-7dee-4201-bf56-33cdaf687f7b admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:52.067606 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59709e-7dee-4201-bf56-33cdaf687f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:52.069015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59709e-7dee-4201-bf56-33cdaf687f7b admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:52.071736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b59709e-7dee-4201-bf56-33cdaf687f7b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:52.075331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b59709e-7dee-4201-bf56-33cdaf687f7b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:52.126773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 879/1752] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:52 2022] POST /identity/v3/projects => generated 383 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:52.126988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c76b2c-1b80-450a-8c61-f60dd5121bab admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:52.150301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c76b2c-1b80-450a-8c61-f60dd5121bab admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:52.180844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca13345-88f3-4149-92a8-a85d4e85fb41 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:52.181237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca13345-88f3-4149-92a8-a85d4e85fb41 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:52.181592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca13345-88f3-4149-92a8-a85d4e85fb41 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:52.183520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 880/1753] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:47:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:47:52.299355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9209650f-ca14-4290-b2a1-065f5b1442ed None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:52.299800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9209650f-ca14-4290-b2a1-065f5b1442ed None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:52.300198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9209650f-ca14-4290-b2a1-065f5b1442ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:52.370461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9209650f-ca14-4290-b2a1-065f5b1442ed None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:52.377890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9209650f-ca14-4290-b2a1-065f5b1442ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:52.411120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc31e79-0289-43ce-9788-7ef986909b7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:52.411761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc31e79-0289-43ce-9788-7ef986909b7f admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:52.412168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc31e79-0289-43ce-9788-7ef986909b7f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:52.412461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc31e79-0289-43ce-9788-7ef986909b7f admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:52.413174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc31e79-0289-43ce-9788-7ef986909b7f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:52.415425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc31e79-0289-43ce-9788-7ef986909b7f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:52.465496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 874/1754] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:52 2022] POST /identity/v3/users => generated 292 bytes in 339 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:52.466439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc31e79-0289-43ce-9788-7ef986909b7f admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:52.492128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc31e79-0289-43ce-9788-7ef986909b7f admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:52.503452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffe89e5-e050-47b8-928d-c9ac3148052a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:52.505726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffe89e5-e050-47b8-928d-c9ac3148052a admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:52.506170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffe89e5-e050-47b8-928d-c9ac3148052a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:52.506660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffe89e5-e050-47b8-928d-c9ac3148052a admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:52.510710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffe89e5-e050-47b8-928d-c9ac3148052a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:52.514993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffe89e5-e050-47b8-928d-c9ac3148052a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:52.543437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 875/1755] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:52 2022] GET /identity/v3/roles => generated 1420 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:47:52.548202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffe89e5-e050-47b8-928d-c9ac3148052a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:52.568176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffe89e5-e050-47b8-928d-c9ac3148052a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:52.592056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ec445-f453-44d2-a4f3-5320bd4a056c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:52.592926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ec445-f453-44d2-a4f3-5320bd4a056c admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:52.593215 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ec445-f453-44d2-a4f3-5320bd4a056c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:52.593524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ec445-f453-44d2-a4f3-5320bd4a056c admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:52.594596 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249ec445-f453-44d2-a4f3-5320bd4a056c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:52.596261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249ec445-f453-44d2-a4f3-5320bd4a056c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:52.622182 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 876/1756] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:52 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:52.628240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ec445-f453-44d2-a4f3-5320bd4a056c admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:52.648872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 881/1757] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:47:52 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 351 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:52.660937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ec445-f453-44d2-a4f3-5320bd4a056c admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:52.670669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1737938-0d71-484d-b061-d477edc4a893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:52.676414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1737938-0d71-484d-b061-d477edc4a893 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:52.676713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1737938-0d71-484d-b061-d477edc4a893 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:52.677001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1737938-0d71-484d-b061-d477edc4a893 admin admin] PATH_INFO: `/v3/projects/0a708949485a4fa1844ceab95079774d/users/e488ec5312a94f0d9b2477bada393303/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:52.724255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1737938-0d71-484d-b061-d477edc4a893 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a708949485a4fa1844ceab95079774d, user_id=e488ec5312a94f0d9b2477bada393303, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:52.727012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1737938-0d71-484d-b061-d477edc4a893 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:52.774604 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe138c4-ca63-4075-b427-fc1be12e40d9 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:52.774973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 877/1758] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:52 2022] PUT /identity/v3/projects/0a708949485a4fa1844ceab95079774d/users/e488ec5312a94f0d9b2477bada393303/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:52.775446 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe138c4-ca63-4075-b427-fc1be12e40d9 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:52.776629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe138c4-ca63-4075-b427-fc1be12e40d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:52.901715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-dfe138c4-ca63-4075-b427-fc1be12e40d9 None None] MFA Rules not processed for user `e488ec5312a94f0d9b2477bada393303`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:52.903170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-dfe138c4-ca63-4075-b427-fc1be12e40d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:53.021080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:53.055721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 882/1759] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:52 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 282 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:53.059917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1737938-0d71-484d-b061-d477edc4a893 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:53.212497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1737938-0d71-484d-b061-d477edc4a893 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:53.218116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1233a8-b1f1-42cc-975f-53b1b2d81442 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:53.218971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1233a8-b1f1-42cc-975f-53b1b2d81442 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:53.219413 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1233a8-b1f1-42cc-975f-53b1b2d81442 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:53.219901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1233a8-b1f1-42cc-975f-53b1b2d81442 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:53.220935 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1233a8-b1f1-42cc-975f-53b1b2d81442 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:53.226489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1233a8-b1f1-42cc-975f-53b1b2d81442 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:53.273145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 878/1760] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:53 2022] POST /identity/v3/projects => generated 383 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:47:53.274580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe138c4-ca63-4075-b427-fc1be12e40d9 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:53.300873 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe138c4-ca63-4075-b427-fc1be12e40d9 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:53.428239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1233a8-b1f1-42cc-975f-53b1b2d81442 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:53.431416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3c1233a8-b1f1-42cc-975f-53b1b2d81442 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:53.437744 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:53.490498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e4fdab-f106-4b81-9edc-aa9b0b68cd5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:53.491231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e4fdab-f106-4b81-9edc-aa9b0b68cd5a admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:53.491587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e4fdab-f106-4b81-9edc-aa9b0b68cd5a admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:53.491939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e4fdab-f106-4b81-9edc-aa9b0b68cd5a admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:53.495499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e4fdab-f106-4b81-9edc-aa9b0b68cd5a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:53.497775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e4fdab-f106-4b81-9edc-aa9b0b68cd5a admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:53.588645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 883/1761] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:53 2022] POST /identity/v3/users => generated 298 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:53.593712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e4fdab-f106-4b81-9edc-aa9b0b68cd5a admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:53.619729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e4fdab-f106-4b81-9edc-aa9b0b68cd5a admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:53.625863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1233a8-b1f1-42cc-975f-53b1b2d81442 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:53.630158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebf23ce-d211-42fa-a7f0-4145e6889be7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:53.631075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebf23ce-d211-42fa-a7f0-4145e6889be7 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:53.631534 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebf23ce-d211-42fa-a7f0-4145e6889be7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:53.631997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebf23ce-d211-42fa-a7f0-4145e6889be7 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:53.633178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebf23ce-d211-42fa-a7f0-4145e6889be7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:53.635932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebf23ce-d211-42fa-a7f0-4145e6889be7 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:53.652621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40416892-9eaa-4791-b6c0-95a6c030dad9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:53.652621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40416892-9eaa-4791-b6c0-95a6c030dad9 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:53.652621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40416892-9eaa-4791-b6c0-95a6c030dad9 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:53.652621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40416892-9eaa-4791-b6c0-95a6c030dad9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:53.668253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 884/1762] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:53.672467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebf23ce-d211-42fa-a7f0-4145e6889be7 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:53.692975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebf23ce-d211-42fa-a7f0-4145e6889be7 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:53.699188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40416892-9eaa-4791-b6c0-95a6c030dad9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:53.700948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e377b021-0947-443b-a025-7538807f40fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:53.701792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e377b021-0947-443b-a025-7538807f40fc admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:53.702079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e377b021-0947-443b-a025-7538807f40fc admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:53.702568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e377b021-0947-443b-a025-7538807f40fc admin admin] PATH_INFO: `/v3/projects/db96db215e1d4743baf316ebe875ddb4/users/714991ba02164f5fb928e96c2d74d02c/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:53.703240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40416892-9eaa-4791-b6c0-95a6c030dad9 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:53.738751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e377b021-0947-443b-a025-7538807f40fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=db96db215e1d4743baf316ebe875ddb4, user_id=714991ba02164f5fb928e96c2d74d02c, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:53.743106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e377b021-0947-443b-a025-7538807f40fc admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:53.771850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 879/1763] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:53 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:53.782376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 885/1764] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:53 2022] PUT /identity/v3/projects/db96db215e1d4743baf316ebe875ddb4/users/714991ba02164f5fb928e96c2d74d02c/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:53.783805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40416892-9eaa-4791-b6c0-95a6c030dad9 service nova] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:53.820334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40416892-9eaa-4791-b6c0-95a6c030dad9 service nova] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:53.911136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373b98e0-6d25-421b-8738-e8253af6a867 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:53.911789 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b98e0-6d25-421b-8738-e8253af6a867 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:53.912061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b98e0-6d25-421b-8738-e8253af6a867 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:53.912340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b98e0-6d25-421b-8738-e8253af6a867 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:53.913075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373b98e0-6d25-421b-8738-e8253af6a867 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:53.914968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373b98e0-6d25-421b-8738-e8253af6a867 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:53.925824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 880/1765] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:53 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:53.930043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e377b021-0947-443b-a025-7538807f40fc admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:53.943160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e377b021-0947-443b-a025-7538807f40fc admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:53.950451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a10f6d5-57e3-44cd-87e0-37323c8e7a16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:53.951564 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a10f6d5-57e3-44cd-87e0-37323c8e7a16 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:53.951564 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a10f6d5-57e3-44cd-87e0-37323c8e7a16 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:53.951928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a10f6d5-57e3-44cd-87e0-37323c8e7a16 admin admin] PATH_INFO: `/v3/projects/db96db215e1d4743baf316ebe875ddb4/users/714991ba02164f5fb928e96c2d74d02c/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:53.955469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a10f6d5-57e3-44cd-87e0-37323c8e7a16 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db96db215e1d4743baf316ebe875ddb4, user_id=714991ba02164f5fb928e96c2d74d02c, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:53.959613 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a10f6d5-57e3-44cd-87e0-37323c8e7a16 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:53.992810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c388dc-8e15-4ba7-b716-e6532c756d61 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:53.992810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c388dc-8e15-4ba7-b716-e6532c756d61 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:53.993081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c388dc-8e15-4ba7-b716-e6532c756d61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:53.997212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 886/1766] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:53 2022] PUT /identity/v3/projects/db96db215e1d4743baf316ebe875ddb4/users/714991ba02164f5fb928e96c2d74d02c/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:47:54.029186 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f8c388dc-8e15-4ba7-b716-e6532c756d61 None None] MFA Rules not processed for user `714991ba02164f5fb928e96c2d74d02c`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:54.030442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f8c388dc-8e15-4ba7-b716-e6532c756d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:54.185259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 881/1767] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:53 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:54.207237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a10f6d5-57e3-44cd-87e0-37323c8e7a16 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:54.214855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-1a10f6d5-57e3-44cd-87e0-37323c8e7a16 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:54.329780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a10f6d5-57e3-44cd-87e0-37323c8e7a16 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:54.343663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a058af-3b49-4d77-bb52-cbdfb8740672 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:54.344274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a058af-3b49-4d77-bb52-cbdfb8740672 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:54.344559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a058af-3b49-4d77-bb52-cbdfb8740672 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:54.344874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a058af-3b49-4d77-bb52-cbdfb8740672 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:54.403326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:54.420201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a058af-3b49-4d77-bb52-cbdfb8740672 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:54.422869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a058af-3b49-4d77-bb52-cbdfb8740672 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:54.447357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 887/1768] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:54.528051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b032423-6a79-4b7f-a7ef-e163e219a410 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:54.529184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b032423-6a79-4b7f-a7ef-e163e219a410 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:54.529631 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b032423-6a79-4b7f-a7ef-e163e219a410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:54.574916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:54.588265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-3b032423-6a79-4b7f-a7ef-e163e219a410 None None] MFA Rules not processed for user `3082a1cb1c0a442ea476d6a8419a1846`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:54.589708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3b032423-6a79-4b7f-a7ef-e163e219a410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:54.719153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 882/1769] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:54 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:54.723555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a058af-3b49-4d77-bb52-cbdfb8740672 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:54.725332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f1a058af-3b49-4d77-bb52-cbdfb8740672 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:54.827711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a058af-3b49-4d77-bb52-cbdfb8740672 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:54.837373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c424a1-7d2c-4d48-b496-9d696133d401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:54.838224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c424a1-7d2c-4d48-b496-9d696133d401 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:54.838690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c424a1-7d2c-4d48-b496-9d696133d401 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:54.838967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c424a1-7d2c-4d48-b496-9d696133d401 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:54.839664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c424a1-7d2c-4d48-b496-9d696133d401 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:54.841981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c424a1-7d2c-4d48-b496-9d696133d401 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:54.876561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 888/1770] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:54 2022] POST /identity/v3/projects => generated 405 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:54.877977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b032423-6a79-4b7f-a7ef-e163e219a410 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:54.891084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b032423-6a79-4b7f-a7ef-e163e219a410 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:54.967444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e155382-97b0-4965-93d3-659caf36b825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:54.968295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e155382-97b0-4965-93d3-659caf36b825 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:54.968588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e155382-97b0-4965-93d3-659caf36b825 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:54.968898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e155382-97b0-4965-93d3-659caf36b825 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:54.969577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e155382-97b0-4965-93d3-659caf36b825 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:54.971509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e155382-97b0-4965-93d3-659caf36b825 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:55.000043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 883/1771] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:54 2022] POST /identity/v3/users => generated 309 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:55.003691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c424a1-7d2c-4d48-b496-9d696133d401 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:55.015229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c424a1-7d2c-4d48-b496-9d696133d401 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:55.020808 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b0ed80-2f12-42d6-9049-712dfcde1520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:55.021677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b0ed80-2f12-42d6-9049-712dfcde1520 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:55.021970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b0ed80-2f12-42d6-9049-712dfcde1520 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:55.022252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b0ed80-2f12-42d6-9049-712dfcde1520 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:55.022989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b0ed80-2f12-42d6-9049-712dfcde1520 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:55.024707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b0ed80-2f12-42d6-9049-712dfcde1520 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:55.036386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 889/1772] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:55 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:47:55.040812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e155382-97b0-4965-93d3-659caf36b825 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:55.053259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e155382-97b0-4965-93d3-659caf36b825 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:55.058160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a01d3-8fb0-4e18-bc29-07c530af0387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:55.058729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a01d3-8fb0-4e18-bc29-07c530af0387 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:55.059018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a01d3-8fb0-4e18-bc29-07c530af0387 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:55.059301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a01d3-8fb0-4e18-bc29-07c530af0387 admin admin] PATH_INFO: `/v3/projects/f27ad64836724894829514d806131a46/users/a7e946eb7882495784b0f4e1a829f70e/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:55.075105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a01d3-8fb0-4e18-bc29-07c530af0387 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f27ad64836724894829514d806131a46, user_id=a7e946eb7882495784b0f4e1a829f70e, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:55.077821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a01d3-8fb0-4e18-bc29-07c530af0387 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:55.090799 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 884/1773] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:55 2022] PUT /identity/v3/projects/f27ad64836724894829514d806131a46/users/a7e946eb7882495784b0f4e1a829f70e/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:55.094775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b0ed80-2f12-42d6-9049-712dfcde1520 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:55.104836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b0ed80-2f12-42d6-9049-712dfcde1520 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:55.160616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07e166e-62ea-4a1c-957a-ec6b7a229303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:55.161323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07e166e-62ea-4a1c-957a-ec6b7a229303 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:55.161607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07e166e-62ea-4a1c-957a-ec6b7a229303 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:55.161885 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07e166e-62ea-4a1c-957a-ec6b7a229303 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:55.162573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07e166e-62ea-4a1c-957a-ec6b7a229303 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:55.164171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07e166e-62ea-4a1c-957a-ec6b7a229303 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:55.173872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 890/1774] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:55 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:47:55.178561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a01d3-8fb0-4e18-bc29-07c530af0387 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:55.188957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a01d3-8fb0-4e18-bc29-07c530af0387 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:55.194495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4751136f-aa16-4fbd-af93-b9fd2347f1f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:55.195187 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4751136f-aa16-4fbd-af93-b9fd2347f1f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:55.195480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4751136f-aa16-4fbd-af93-b9fd2347f1f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:55.195763 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4751136f-aa16-4fbd-af93-b9fd2347f1f5 admin admin] PATH_INFO: `/v3/projects/f27ad64836724894829514d806131a46/users/a7e946eb7882495784b0f4e1a829f70e/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:55.197801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4751136f-aa16-4fbd-af93-b9fd2347f1f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f27ad64836724894829514d806131a46, user_id=a7e946eb7882495784b0f4e1a829f70e, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:55.200495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4751136f-aa16-4fbd-af93-b9fd2347f1f5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:55.211444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 885/1775] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:55 2022] PUT /identity/v3/projects/f27ad64836724894829514d806131a46/users/a7e946eb7882495784b0f4e1a829f70e/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:56.104379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d851a-3294-4b9c-baf6-078fb44041d2 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:56.104930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d851a-3294-4b9c-baf6-078fb44041d2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:56.105030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc6d851a-3294-4b9c-baf6-078fb44041d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:56.155430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-cc6d851a-3294-4b9c-baf6-078fb44041d2 None None] MFA Rules not processed for user `a7e946eb7882495784b0f4e1a829f70e`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:56.156783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-cc6d851a-3294-4b9c-baf6-078fb44041d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:56.227302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:56.267078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 891/1776] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:56 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:57.057147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4751136f-aa16-4fbd-af93-b9fd2347f1f5 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:57.058925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4751136f-aa16-4fbd-af93-b9fd2347f1f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:57.082080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f4fc1-a175-4b7a-b4f4-8c0e38d016c4 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:57.083247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f4fc1-a175-4b7a-b4f4-8c0e38d016c4 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:57.083247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396f4fc1-a175-4b7a-b4f4-8c0e38d016c4 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:57.084909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 892/1777] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:47:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:47:57.123593 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:57.127854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd33bcd8-9f20-47f5-bbb4-27617a920cf7 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:57.128921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd33bcd8-9f20-47f5-bbb4-27617a920cf7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:57.129870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd33bcd8-9f20-47f5-bbb4-27617a920cf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:57.154986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4751136f-aa16-4fbd-af93-b9fd2347f1f5 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:57.161572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:57.162325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:57.163397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:57.163397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] PATH_INFO: `/v3/users/50203897c78c41528973f0eeaf879d56` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:57.164831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=50203897c78c41528973f0eeaf879d56)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:57.169833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-fd33bcd8-9f20-47f5-bbb4-27617a920cf7 None None] MFA Rules not processed for user `e3da5e428049419a9b35494655f52484`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:57.172246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fd33bcd8-9f20-47f5-bbb4-27617a920cf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:57.177253 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:57.257300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '50203897c78c41528973f0eeaf879d56'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:57.268586 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '50203897c78c41528973f0eeaf879d56'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:57.269888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '50203897c78c41528973f0eeaf879d56'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:57.270516 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '50203897c78c41528973f0eeaf879d56'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:57.280169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '50203897c78c41528973f0eeaf879d56'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:57.302371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 893/1778] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:47:57 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:57.311225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 886/1779] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:57 2022] DELETE /identity/v3/users/50203897c78c41528973f0eeaf879d56 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:57.319106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9691d2e0-5353-4a4c-90c8-88b9b973b081 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:57.319598 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9691d2e0-5353-4a4c-90c8-88b9b973b081 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:57.320075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9691d2e0-5353-4a4c-90c8-88b9b973b081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:57.329267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:57.343340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:57.349048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9691d2e0-5353-4a4c-90c8-88b9b973b081 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:57.350284 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9691d2e0-5353-4a4c-90c8-88b9b973b081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:57.426479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c513725d-780b-43f4-8f40-404a7957b5c9 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:57.431902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4281c55f-9b47-4c64-aaa2-adbc68e6cb61 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:57.432545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281c55f-9b47-4c64-aaa2-adbc68e6cb61 service placement] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:57.432844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 894/1780] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:57.433070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281c55f-9b47-4c64-aaa2-adbc68e6cb61 service placement] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:57.433172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281c55f-9b47-4c64-aaa2-adbc68e6cb61 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:57.441222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9691d2e0-5353-4a4c-90c8-88b9b973b081 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:57.443437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9691d2e0-5353-4a4c-90c8-88b9b973b081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:57.445477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4281c55f-9b47-4c64-aaa2-adbc68e6cb61 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:57.447490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4281c55f-9b47-4c64-aaa2-adbc68e6cb61 service placement] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:57.461619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 887/1781] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:57 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:47:57.522449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9691d2e0-5353-4a4c-90c8-88b9b973b081 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:57.526126 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae236e4-0f71-4fd3-a91b-7db9b60b1aa9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:57.526762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae236e4-0f71-4fd3-a91b-7db9b60b1aa9 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:57.527134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae236e4-0f71-4fd3-a91b-7db9b60b1aa9 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:57.527440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae236e4-0f71-4fd3-a91b-7db9b60b1aa9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:57.537121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae236e4-0f71-4fd3-a91b-7db9b60b1aa9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:57.538540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae236e4-0f71-4fd3-a91b-7db9b60b1aa9 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:57.551451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 895/1782] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:47:57.992228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793cea58-271f-41f1-88f9-eb6a8d6552e7 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:57.992782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793cea58-271f-41f1-88f9-eb6a8d6552e7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:57.993171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793cea58-271f-41f1-88f9-eb6a8d6552e7 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:57.994875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 888/1783] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:47:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:47:58.031435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d22124a-29c3-4b2e-be21-d376cf19113d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:58.032343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d22124a-29c3-4b2e-be21-d376cf19113d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:58.032343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d22124a-29c3-4b2e-be21-d376cf19113d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:58.055122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9d22124a-29c3-4b2e-be21-d376cf19113d None None] MFA Rules not processed for user `e3da5e428049419a9b35494655f52484`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:58.056591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9d22124a-29c3-4b2e-be21-d376cf19113d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:58.075032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793cea58-271f-41f1-88f9-eb6a8d6552e7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:58.075496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 896/1784] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:47:58 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:58.078194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-793cea58-271f-41f1-88f9-eb6a8d6552e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:58.093862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d22124a-29c3-4b2e-be21-d376cf19113d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:58.104965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793cea58-271f-41f1-88f9-eb6a8d6552e7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:58.108852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d22124a-29c3-4b2e-be21-d376cf19113d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:58.110894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:58.111825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:58.112296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:58.112590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] PATH_INFO: `/v3/users/3082a1cb1c0a442ea476d6a8419a1846` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:58.114728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] RBAC: Authorizing `identity:delete_user(user_id=3082a1cb1c0a442ea476d6a8419a1846)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:58.116143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89be54ff-7140-4528-80bd-9544a2aa6fa8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:58.116804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89be54ff-7140-4528-80bd-9544a2aa6fa8 service placement] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:58.117106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89be54ff-7140-4528-80bd-9544a2aa6fa8 service placement] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:58.117427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89be54ff-7140-4528-80bd-9544a2aa6fa8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:58.119029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:58.133477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89be54ff-7140-4528-80bd-9544a2aa6fa8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:58.135336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89be54ff-7140-4528-80bd-9544a2aa6fa8 service placement] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:58.152208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 897/1785] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:47:58 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:47:58.153246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89be54ff-7140-4528-80bd-9544a2aa6fa8 service placement] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:58.155137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-89be54ff-7140-4528-80bd-9544a2aa6fa8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:58.165524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:58.175566 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89be54ff-7140-4528-80bd-9544a2aa6fa8 service placement] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:58.180772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:58.181428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:58.181722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:58.182141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] PATH_INFO: `/v3/projects/95e9cb3f60694f9aaffa1b454f5a14e7` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:58.183390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] RBAC: Authorizing `identity:delete_project(project_id=95e9cb3f60694f9aaffa1b454f5a14e7)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:58.183878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3082a1cb1c0a442ea476d6a8419a1846'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.186908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:58.193671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3082a1cb1c0a442ea476d6a8419a1846'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.194492 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3082a1cb1c0a442ea476d6a8419a1846'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.195117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3082a1cb1c0a442ea476d6a8419a1846'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.196589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:58.206166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3082a1cb1c0a442ea476d6a8419a1846'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.225774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 889/1786] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:58 2022] DELETE /identity/v3/users/3082a1cb1c0a442ea476d6a8419a1846 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:47:58.237167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95e9cb3f60694f9aaffa1b454f5a14e7', 'request_id': 'req-89765d63-cc94-4c09-8664-1ec038947ded'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.244075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95e9cb3f60694f9aaffa1b454f5a14e7', 'request_id': 'req-89765d63-cc94-4c09-8664-1ec038947ded'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.251936 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95e9cb3f60694f9aaffa1b454f5a14e7', 'request_id': 'req-89765d63-cc94-4c09-8664-1ec038947ded'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.264725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] The token cache is being invalidate because project 95e9cb3f60694f9aaffa1b454f5a14e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:58.265006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.265672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.267222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 898/1787] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:58 2022] DELETE /identity/v3/projects/95e9cb3f60694f9aaffa1b454f5a14e7 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:58.271758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:58.273411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:58.347408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb27d61d-e0fc-4288-88fd-0aff2f0d952a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:58.352758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:58.353495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:58.353925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:58.354259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] PATH_INFO: `/v3/users/115d1dea431849b6a1a2fe7c83957d76` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:58.355698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] RBAC: Authorizing `identity:delete_user(user_id=115d1dea431849b6a1a2fe7c83957d76)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:58.358561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:58.427566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '115d1dea431849b6a1a2fe7c83957d76'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.435632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '115d1dea431849b6a1a2fe7c83957d76'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.437685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '115d1dea431849b6a1a2fe7c83957d76'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.438457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '115d1dea431849b6a1a2fe7c83957d76'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.450063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '115d1dea431849b6a1a2fe7c83957d76'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.472851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 890/1788] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:58 2022] DELETE /identity/v3/users/115d1dea431849b6a1a2fe7c83957d76 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:58.529734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:58.531371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:58.634881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89765d63-cc94-4c09-8664-1ec038947ded admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:58.701316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:58.702519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:58.702818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:58.703105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] PATH_INFO: `/v3/projects/f51e2ccad2b749a4bc199e678510b53f` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:58.704816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] RBAC: Authorizing `identity:delete_project(project_id=f51e2ccad2b749a4bc199e678510b53f)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:58.706733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:58.757557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:58.769561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f51e2ccad2b749a4bc199e678510b53f', 'request_id': 'req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.776937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f51e2ccad2b749a4bc199e678510b53f', 'request_id': 'req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.786251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f51e2ccad2b749a4bc199e678510b53f', 'request_id': 'req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.797625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] The token cache is being invalidate because project f51e2ccad2b749a4bc199e678510b53f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:58.797926 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.798612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:58.801644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 899/1789] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:58 2022] DELETE /identity/v3/projects/f51e2ccad2b749a4bc199e678510b53f => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:58.808171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:58.818869 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:58.835825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:58.837432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:58.840360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765abb50-1c0a-42e7-a539-4af71574a802 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:58.935443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:58.941214 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bb28c7-1c49-4fed-a6ec-b8d79ae46dbf admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:58.948932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:58.949473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:58.949883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:58.950391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:58.950497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:58.950680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] PATH_INFO: `/v3/projects/8f06f68885a54b40b7db2cb48e93e1d3` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:58.950992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:58.951373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] PATH_INFO: `/v3/users/a7e946eb7882495784b0f4e1a829f70e` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:58.952638 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f06f68885a54b40b7db2cb48e93e1d3)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:58.953378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7e946eb7882495784b0f4e1a829f70e)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:58.953961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:58.955646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.012864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f06f68885a54b40b7db2cb48e93e1d3', 'request_id': 'req-5ba72a43-470a-4486-b685-410d9ac6207f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.013070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7e946eb7882495784b0f4e1a829f70e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.021506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f06f68885a54b40b7db2cb48e93e1d3', 'request_id': 'req-5ba72a43-470a-4486-b685-410d9ac6207f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.021697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7e946eb7882495784b0f4e1a829f70e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.022379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7e946eb7882495784b0f4e1a829f70e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.023004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7e946eb7882495784b0f4e1a829f70e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.031267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4d4808ba-14c1-4266-824a-80f178541fa3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7e946eb7882495784b0f4e1a829f70e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.038605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f06f68885a54b40b7db2cb48e93e1d3', 'request_id': 'req-5ba72a43-470a-4486-b685-410d9ac6207f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.052187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] The token cache is being invalidate because project 8f06f68885a54b40b7db2cb48e93e1d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:59.052576 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.053387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5ba72a43-470a-4486-b685-410d9ac6207f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.054277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 891/1790] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:47:58 2022] DELETE /identity/v3/users/a7e946eb7882495784b0f4e1a829f70e => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:59.054956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 900/1791] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:58 2022] DELETE /identity/v3/projects/8f06f68885a54b40b7db2cb48e93e1d3 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:59.060936 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf389a6b-2f27-418e-9c35-088433ac2612 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.061261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf389a6b-2f27-418e-9c35-088433ac2612 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.061551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf389a6b-2f27-418e-9c35-088433ac2612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.082546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-bf389a6b-2f27-418e-9c35-088433ac2612 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:59.083773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-bf389a6b-2f27-418e-9c35-088433ac2612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:59.149315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 892/1792] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:59.156953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c8b65f-de5c-49ab-a2d2-f3eba729d70b None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.157465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c8b65f-de5c-49ab-a2d2-f3eba729d70b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.157637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c8b65f-de5c-49ab-a2d2-f3eba729d70b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.178451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-85c8b65f-de5c-49ab-a2d2-f3eba729d70b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:59.179760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-85c8b65f-de5c-49ab-a2d2-f3eba729d70b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:59.187762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 901/1793] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:59.193532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf389a6b-2f27-418e-9c35-088433ac2612 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.203173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf389a6b-2f27-418e-9c35-088433ac2612 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.208324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcb8119-5f97-40ad-b4f8-0ba9d5ac1aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.208979 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcb8119-5f97-40ad-b4f8-0ba9d5ac1aa0 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.209268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcb8119-5f97-40ad-b4f8-0ba9d5ac1aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.209549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcb8119-5f97-40ad-b4f8-0ba9d5ac1aa0 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.210255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcb8119-5f97-40ad-b4f8-0ba9d5ac1aa0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:59.210565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcb8119-5f97-40ad-b4f8-0ba9d5ac1aa0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.212168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcb8119-5f97-40ad-b4f8-0ba9d5ac1aa0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.225371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 893/1794] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:59.230044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c8b65f-de5c-49ab-a2d2-f3eba729d70b None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.236399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcb8119-5f97-40ad-b4f8-0ba9d5ac1aa0 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.238370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3fcb8119-5f97-40ad-b4f8-0ba9d5ac1aa0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:59.240706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c8b65f-de5c-49ab-a2d2-f3eba729d70b None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.246517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41143f35-593d-404f-a3c8-6d5480b2ddd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.247152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41143f35-593d-404f-a3c8-6d5480b2ddd7 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.247447 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41143f35-593d-404f-a3c8-6d5480b2ddd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.247722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41143f35-593d-404f-a3c8-6d5480b2ddd7 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.248620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41143f35-593d-404f-a3c8-6d5480b2ddd7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.253415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcb8119-5f97-40ad-b4f8-0ba9d5ac1aa0 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.254741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41143f35-593d-404f-a3c8-6d5480b2ddd7 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.259410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.260097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.260383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.260689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] PATH_INFO: `/v3/projects/f27ad64836724894829514d806131a46` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.261872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f27ad64836724894829514d806131a46)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.263804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.287267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 902/1795] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/projects => generated 363 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:47:59.292277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41143f35-593d-404f-a3c8-6d5480b2ddd7 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.319416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f27ad64836724894829514d806131a46', 'request_id': 'req-749e374f-5e11-4495-abf2-4d3b294e9065'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.321950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:59.324261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:59.328094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f27ad64836724894829514d806131a46', 'request_id': 'req-749e374f-5e11-4495-abf2-4d3b294e9065'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.335675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f27ad64836724894829514d806131a46', 'request_id': 'req-749e374f-5e11-4495-abf2-4d3b294e9065'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.347294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] The token cache is being invalidate because project f27ad64836724894829514d806131a46 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:47:59.347588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.349422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-749e374f-5e11-4495-abf2-4d3b294e9065 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:47:59.351003 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 894/1796] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:47:59 2022] DELETE /identity/v3/projects/f27ad64836724894829514d806131a46 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:59.355777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83699f20-423c-4b85-8a39-8055dd09522e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.356092 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83699f20-423c-4b85-8a39-8055dd09522e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.356370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83699f20-423c-4b85-8a39-8055dd09522e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.367475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41143f35-593d-404f-a3c8-6d5480b2ddd7 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.376210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-83699f20-423c-4b85-8a39-8055dd09522e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:59.377512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-83699f20-423c-4b85-8a39-8055dd09522e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:59.437617 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd2a483-3b65-4db7-bbc4-599c71992cb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.438646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd2a483-3b65-4db7-bbc4-599c71992cb4 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.439120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd2a483-3b65-4db7-bbc4-599c71992cb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.439597 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd2a483-3b65-4db7-bbc4-599c71992cb4 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.440575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd2a483-3b65-4db7-bbc4-599c71992cb4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.444087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd2a483-3b65-4db7-bbc4-599c71992cb4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.456936 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 895/1797] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:59.465327 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce55b66-fd9f-4589-b418-0acd4151aad2 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.465620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce55b66-fd9f-4589-b418-0acd4151aad2 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.465957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce55b66-fd9f-4589-b418-0acd4151aad2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.471711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 903/1798] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/users => generated 282 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:59.476353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd2a483-3b65-4db7-bbc4-599c71992cb4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.478075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8cd2a483-3b65-4db7-bbc4-599c71992cb4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:59.486406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-3ce55b66-fd9f-4589-b418-0acd4151aad2 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:59.487590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3ce55b66-fd9f-4589-b418-0acd4151aad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:59.490579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd2a483-3b65-4db7-bbc4-599c71992cb4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.495261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63b1b07-a64d-4605-bad4-d903ddb6151c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.495859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63b1b07-a64d-4605-bad4-d903ddb6151c admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.496149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63b1b07-a64d-4605-bad4-d903ddb6151c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.496431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63b1b07-a64d-4605-bad4-d903ddb6151c admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.496615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 896/1799] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:47:59.497529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63b1b07-a64d-4605-bad4-d903ddb6151c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.499166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63b1b07-a64d-4605-bad4-d903ddb6151c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.500461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce55b66-fd9f-4589-b418-0acd4151aad2 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.509577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce55b66-fd9f-4589-b418-0acd4151aad2 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.512824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 904/1800] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:59 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:47:59.513747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63b1b07-a64d-4605-bad4-d903ddb6151c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.516494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5550da-a03d-4b90-a86c-9b3492614ded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.517161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5550da-a03d-4b90-a86c-9b3492614ded admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.517453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5550da-a03d-4b90-a86c-9b3492614ded admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.517733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5550da-a03d-4b90-a86c-9b3492614ded admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.518437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5550da-a03d-4b90-a86c-9b3492614ded admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:47:59.518745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5550da-a03d-4b90-a86c-9b3492614ded admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.520417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5550da-a03d-4b90-a86c-9b3492614ded admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.526067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63b1b07-a64d-4605-bad4-d903ddb6151c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.532798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72462c0e-b11e-4a27-a31d-12e4e9600f54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.536968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 897/1801] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:47:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:47:59.537112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72462c0e-b11e-4a27-a31d-12e4e9600f54 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.537112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72462c0e-b11e-4a27-a31d-12e4e9600f54 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.537112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72462c0e-b11e-4a27-a31d-12e4e9600f54 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.537112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72462c0e-b11e-4a27-a31d-12e4e9600f54 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.537727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72462c0e-b11e-4a27-a31d-12e4e9600f54 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.538430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5550da-a03d-4b90-a86c-9b3492614ded admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.555765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 905/1802] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:59 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:47:59.555934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5550da-a03d-4b90-a86c-9b3492614ded admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.557006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72462c0e-b11e-4a27-a31d-12e4e9600f54 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.568275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba6c567-103f-4d0c-83b0-9f7718fc49a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.569320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72462c0e-b11e-4a27-a31d-12e4e9600f54 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.570825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6c567-103f-4d0c-83b0-9f7718fc49a2 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.571194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6c567-103f-4d0c-83b0-9f7718fc49a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.571546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba6c567-103f-4d0c-83b0-9f7718fc49a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.572336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba6c567-103f-4d0c-83b0-9f7718fc49a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.574356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba6c567-103f-4d0c-83b0-9f7718fc49a2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.574644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf13f0a-7a53-4914-83a0-8b0e1e30aec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.575582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf13f0a-7a53-4914-83a0-8b0e1e30aec7 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.575874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf13f0a-7a53-4914-83a0-8b0e1e30aec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.576151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf13f0a-7a53-4914-83a0-8b0e1e30aec7 admin admin] PATH_INFO: `/v3/projects/a49ff0f2869741fb914ec3fbcd786b1f/users/f50ab8be666243708e1ad10889cdaba3/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.594752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf13f0a-7a53-4914-83a0-8b0e1e30aec7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a49ff0f2869741fb914ec3fbcd786b1f, user_id=f50ab8be666243708e1ad10889cdaba3, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.600885 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf13f0a-7a53-4914-83a0-8b0e1e30aec7 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.610173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 898/1803] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/projects => generated 363 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:59.614484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba6c567-103f-4d0c-83b0-9f7718fc49a2 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.618416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 906/1804] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:59 2022] PUT /identity/v3/projects/a49ff0f2869741fb914ec3fbcd786b1f/users/f50ab8be666243708e1ad10889cdaba3/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:47:59.629730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2869a4-95e3-4f28-9546-011b72cfcb6c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.630045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2869a4-95e3-4f28-9546-011b72cfcb6c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.630404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2869a4-95e3-4f28-9546-011b72cfcb6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.690133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2b2869a4-95e3-4f28-9546-011b72cfcb6c None None] MFA Rules not processed for user `f50ab8be666243708e1ad10889cdaba3`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:47:59.692072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2b2869a4-95e3-4f28-9546-011b72cfcb6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:47:59.705401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba6c567-103f-4d0c-83b0-9f7718fc49a2 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.765641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:59.782550 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 907/1805] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:47:59.788767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2869a4-95e3-4f28-9546-011b72cfcb6c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.794836 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d898a1ff-711c-482b-9f99-a3de29daba14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.795512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d898a1ff-711c-482b-9f99-a3de29daba14 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.795804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d898a1ff-711c-482b-9f99-a3de29daba14 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.796198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:47:59.796423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d898a1ff-711c-482b-9f99-a3de29daba14 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.797164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d898a1ff-711c-482b-9f99-a3de29daba14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.799423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d898a1ff-711c-482b-9f99-a3de29daba14 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.828641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 899/1806] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/users => generated 282 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:47:59.831117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d898a1ff-711c-482b-9f99-a3de29daba14 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.841242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d898a1ff-711c-482b-9f99-a3de29daba14 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.846270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca699ce-f8b6-4afb-820a-7c638bb3c63e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.846946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca699ce-f8b6-4afb-820a-7c638bb3c63e admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.847242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca699ce-f8b6-4afb-820a-7c638bb3c63e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.847529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca699ce-f8b6-4afb-820a-7c638bb3c63e admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.848896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca699ce-f8b6-4afb-820a-7c638bb3c63e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.850629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca699ce-f8b6-4afb-820a-7c638bb3c63e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.860752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 900/1807] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:47:59.862373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2869a4-95e3-4f28-9546-011b72cfcb6c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.864399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca699ce-f8b6-4afb-820a-7c638bb3c63e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.867485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70230ebc-c8e9-4d75-945d-2624d135c026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.868114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70230ebc-c8e9-4d75-945d-2624d135c026 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.868406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70230ebc-c8e9-4d75-945d-2624d135c026 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.868768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70230ebc-c8e9-4d75-945d-2624d135c026 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.869453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70230ebc-c8e9-4d75-945d-2624d135c026 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.871364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70230ebc-c8e9-4d75-945d-2624d135c026 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.873877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca699ce-f8b6-4afb-820a-7c638bb3c63e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.878458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011a7e22-0299-41d7-89ff-802deffd1ff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.879083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011a7e22-0299-41d7-89ff-802deffd1ff9 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.879374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011a7e22-0299-41d7-89ff-802deffd1ff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.879656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011a7e22-0299-41d7-89ff-802deffd1ff9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.880397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011a7e22-0299-41d7-89ff-802deffd1ff9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.881945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011a7e22-0299-41d7-89ff-802deffd1ff9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.893047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 901/1808] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:47:59 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:47:59.897837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011a7e22-0299-41d7-89ff-802deffd1ff9 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.900655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 908/1809] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/projects => generated 363 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:47:59.904305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70230ebc-c8e9-4d75-945d-2624d135c026 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:47:59.908378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011a7e22-0299-41d7-89ff-802deffd1ff9 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.976212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70230ebc-c8e9-4d75-945d-2624d135c026 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:47:59.978065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0114ae78-94f8-47c6-be2c-c3195fd4b319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.978753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0114ae78-94f8-47c6-be2c-c3195fd4b319 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.979037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0114ae78-94f8-47c6-be2c-c3195fd4b319 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.979316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0114ae78-94f8-47c6-be2c-c3195fd4b319 admin admin] PATH_INFO: `/v3/projects/d86274d381754746bd01fbd52bb1ae57/users/e4464812190545f286efa68de4ba72c1/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.980899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161c99d8-0425-4b29-a216-e2a28875fc6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:47:59.981531 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161c99d8-0425-4b29-a216-e2a28875fc6c admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:47:59.981818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161c99d8-0425-4b29-a216-e2a28875fc6c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:47:59.982097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161c99d8-0425-4b29-a216-e2a28875fc6c admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:47:59.982924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161c99d8-0425-4b29-a216-e2a28875fc6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.984918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161c99d8-0425-4b29-a216-e2a28875fc6c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:47:59.996642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0114ae78-94f8-47c6-be2c-c3195fd4b319 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d86274d381754746bd01fbd52bb1ae57, user_id=e4464812190545f286efa68de4ba72c1, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:47:59.999747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0114ae78-94f8-47c6-be2c-c3195fd4b319 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.013154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 909/1810] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:47:59 2022] POST /identity/v3/users => generated 288 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:00.014476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161c99d8-0425-4b29-a216-e2a28875fc6c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.014827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 902/1811] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:47:59 2022] PUT /identity/v3/projects/d86274d381754746bd01fbd52bb1ae57/users/e4464812190545f286efa68de4ba72c1/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:00.025549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161c99d8-0425-4b29-a216-e2a28875fc6c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.026440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa91ef-ca48-4226-9e46-36dad3d48bab None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.026737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa91ef-ca48-4226-9e46-36dad3d48bab None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.027020 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aa91ef-ca48-4226-9e46-36dad3d48bab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.056406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-40aa91ef-ca48-4226-9e46-36dad3d48bab None None] MFA Rules not processed for user `e4464812190545f286efa68de4ba72c1`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:00.058173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-40aa91ef-ca48-4226-9e46-36dad3d48bab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:00.094393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7800bc-25af-4249-b8b1-541412eef52d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.095407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7800bc-25af-4249-b8b1-541412eef52d admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.095883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7800bc-25af-4249-b8b1-541412eef52d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.096367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7800bc-25af-4249-b8b1-541412eef52d admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.097649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7800bc-25af-4249-b8b1-541412eef52d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.100628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7800bc-25af-4249-b8b1-541412eef52d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.118042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 910/1812] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:00 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:00.120830 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7800bc-25af-4249-b8b1-541412eef52d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.134493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7800bc-25af-4249-b8b1-541412eef52d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.140579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8906166-c9ca-4d39-9fab-113c5829b4ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.141287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8906166-c9ca-4d39-9fab-113c5829b4ca admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.141578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8906166-c9ca-4d39-9fab-113c5829b4ca admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.141867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8906166-c9ca-4d39-9fab-113c5829b4ca admin admin] PATH_INFO: `/v3/projects/6673e4bdeffb40aca77e2740b415c15b/users/707564ea64354507a0efb1aee1801de4/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.150981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 903/1813] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:00 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:00.155426 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aa91ef-ca48-4226-9e46-36dad3d48bab None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.159644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8906166-c9ca-4d39-9fab-113c5829b4ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=6673e4bdeffb40aca77e2740b415c15b, user_id=707564ea64354507a0efb1aee1801de4, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.164454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8906166-c9ca-4d39-9fab-113c5829b4ca admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.166593 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aa91ef-ca48-4226-9e46-36dad3d48bab None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.167137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:00.172372 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984ac951-3853-44be-b8f0-e1885c443346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.173072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984ac951-3853-44be-b8f0-e1885c443346 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.173593 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984ac951-3853-44be-b8f0-e1885c443346 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.173996 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984ac951-3853-44be-b8f0-e1885c443346 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.174666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984ac951-3853-44be-b8f0-e1885c443346 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.176855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984ac951-3853-44be-b8f0-e1885c443346 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.183318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 911/1814] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:00 2022] PUT /identity/v3/projects/6673e4bdeffb40aca77e2740b415c15b/users/707564ea64354507a0efb1aee1801de4/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:00.187744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8906166-c9ca-4d39-9fab-113c5829b4ca admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.209357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 904/1815] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:00 2022] POST /identity/v3/projects => generated 363 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:00.217594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984ac951-3853-44be-b8f0-e1885c443346 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.224283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:00.260102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8906166-c9ca-4d39-9fab-113c5829b4ca admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.282469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984ac951-3853-44be-b8f0-e1885c443346 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.288411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4cdfdb-ffc4-4622-9219-0e222ecce8bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.289084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4cdfdb-ffc4-4622-9219-0e222ecce8bc admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.289368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4cdfdb-ffc4-4622-9219-0e222ecce8bc admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.289646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4cdfdb-ffc4-4622-9219-0e222ecce8bc admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.290430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4cdfdb-ffc4-4622-9219-0e222ecce8bc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.292379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4cdfdb-ffc4-4622-9219-0e222ecce8bc admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.321185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 905/1816] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:00 2022] POST /identity/v3/users => generated 288 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:00.321378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f3512d-a1b6-463d-9d94-aec886d2c34e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.321967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f3512d-a1b6-463d-9d94-aec886d2c34e admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.322227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4cdfdb-ffc4-4622-9219-0e222ecce8bc admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.322331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f3512d-a1b6-463d-9d94-aec886d2c34e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.322522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f3512d-a1b6-463d-9d94-aec886d2c34e admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.323323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f3512d-a1b6-463d-9d94-aec886d2c34e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.325013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f3512d-a1b6-463d-9d94-aec886d2c34e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.332574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4cdfdb-ffc4-4622-9219-0e222ecce8bc admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.335977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 912/1817] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:00 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:00.337418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6028ce00-1872-42a4-92a5-f5e68da83110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.338027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6028ce00-1872-42a4-92a5-f5e68da83110 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.338314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6028ce00-1872-42a4-92a5-f5e68da83110 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.338769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6028ce00-1872-42a4-92a5-f5e68da83110 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.340260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f3512d-a1b6-463d-9d94-aec886d2c34e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.340373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6028ce00-1872-42a4-92a5-f5e68da83110 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.342436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6028ce00-1872-42a4-92a5-f5e68da83110 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.351512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f3512d-a1b6-463d-9d94-aec886d2c34e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.357738 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 906/1818] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:00.357877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a17a25-a79b-4d21-a3dd-0a48851e9d6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.357877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a17a25-a79b-4d21-a3dd-0a48851e9d6c admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.358070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a17a25-a79b-4d21-a3dd-0a48851e9d6c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.358181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a17a25-a79b-4d21-a3dd-0a48851e9d6c admin admin] PATH_INFO: `/v3/projects/6673e4bdeffb40aca77e2740b415c15b/users/707564ea64354507a0efb1aee1801de4/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.360261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a17a25-a79b-4d21-a3dd-0a48851e9d6c admin admin] RBAC: Authorizing `identity:create_grant(project_id=6673e4bdeffb40aca77e2740b415c15b, user_id=707564ea64354507a0efb1aee1801de4, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.360764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6028ce00-1872-42a4-92a5-f5e68da83110 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.362966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a17a25-a79b-4d21-a3dd-0a48851e9d6c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.370792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6028ce00-1872-42a4-92a5-f5e68da83110 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.374222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 913/1819] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:00 2022] PUT /identity/v3/projects/6673e4bdeffb40aca77e2740b415c15b/users/707564ea64354507a0efb1aee1801de4/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:00.385568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e7b859-ea18-4919-9662-4198f664b717 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.385787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e7b859-ea18-4919-9662-4198f664b717 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.386068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e7b859-ea18-4919-9662-4198f664b717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.418051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-93e7b859-ea18-4919-9662-4198f664b717 None None] MFA Rules not processed for user `707564ea64354507a0efb1aee1801de4`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:00.419616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-93e7b859-ea18-4919-9662-4198f664b717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:00.442867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d164c1b0-010c-4c3b-8247-6838d4a7969a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.443583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d164c1b0-010c-4c3b-8247-6838d4a7969a admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.443857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d164c1b0-010c-4c3b-8247-6838d4a7969a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.444135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d164c1b0-010c-4c3b-8247-6838d4a7969a admin admin] PATH_INFO: `/v3/projects/6f8ee233f672493a95ce967e6e828fcf/users/2001f82027274d1dbf0280ea7d1610a6/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.461829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d164c1b0-010c-4c3b-8247-6838d4a7969a admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f8ee233f672493a95ce967e6e828fcf, user_id=2001f82027274d1dbf0280ea7d1610a6, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.464548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d164c1b0-010c-4c3b-8247-6838d4a7969a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.478516 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 907/1820] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:00 2022] PUT /identity/v3/projects/6f8ee233f672493a95ce967e6e828fcf/users/2001f82027274d1dbf0280ea7d1610a6/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:00.482842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d164c1b0-010c-4c3b-8247-6838d4a7969a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.491421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 914/1821] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:00 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:00.502757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e7b859-ea18-4919-9662-4198f664b717 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.504643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-93e7b859-ea18-4919-9662-4198f664b717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:00.514152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:00.547401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d164c1b0-010c-4c3b-8247-6838d4a7969a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.552318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bab34b-9b41-499a-9434-56e35b1e0ca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.552990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bab34b-9b41-499a-9434-56e35b1e0ca3 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.553279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bab34b-9b41-499a-9434-56e35b1e0ca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.553557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bab34b-9b41-499a-9434-56e35b1e0ca3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.554247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bab34b-9b41-499a-9434-56e35b1e0ca3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.555838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bab34b-9b41-499a-9434-56e35b1e0ca3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.564228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e7b859-ea18-4919-9662-4198f664b717 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.569096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 908/1822] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:00 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:00.570803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bab34b-9b41-499a-9434-56e35b1e0ca3 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.575704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:00.580780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bab34b-9b41-499a-9434-56e35b1e0ca3 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.592912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f491683-f70b-4a52-a0b5-e577e73c7520 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.593594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f491683-f70b-4a52-a0b5-e577e73c7520 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.593876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f491683-f70b-4a52-a0b5-e577e73c7520 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.594194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f491683-f70b-4a52-a0b5-e577e73c7520 admin admin] PATH_INFO: `/v3/projects/6f8ee233f672493a95ce967e6e828fcf/users/2001f82027274d1dbf0280ea7d1610a6/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.596164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f491683-f70b-4a52-a0b5-e577e73c7520 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f8ee233f672493a95ce967e6e828fcf, user_id=2001f82027274d1dbf0280ea7d1610a6, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.599563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f491683-f70b-4a52-a0b5-e577e73c7520 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.610429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 909/1823] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:00 2022] PUT /identity/v3/projects/6f8ee233f672493a95ce967e6e828fcf/users/2001f82027274d1dbf0280ea7d1610a6/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:00.619363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93523f06-cb83-4998-8e1a-941172368454 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.619363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93523f06-cb83-4998-8e1a-941172368454 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.619363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93523f06-cb83-4998-8e1a-941172368454 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.619363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93523f06-cb83-4998-8e1a-941172368454 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.620780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31e1b5e-d95e-4841-ae89-4154b6dd81cf None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.621078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31e1b5e-d95e-4841-ae89-4154b6dd81cf None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.621359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31e1b5e-d95e-4841-ae89-4154b6dd81cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.653139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-b31e1b5e-d95e-4841-ae89-4154b6dd81cf None None] MFA Rules not processed for user `2001f82027274d1dbf0280ea7d1610a6`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:00.653139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b31e1b5e-d95e-4841-ae89-4154b6dd81cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:00.685533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93523f06-cb83-4998-8e1a-941172368454 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.687028 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93523f06-cb83-4998-8e1a-941172368454 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.697993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 915/1824] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:00 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:00.722937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 910/1825] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:00 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:00.735163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93523f06-cb83-4998-8e1a-941172368454 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.736593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-93523f06-cb83-4998-8e1a-941172368454 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:00.748220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93523f06-cb83-4998-8e1a-941172368454 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.778815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31e1b5e-d95e-4841-ae89-4154b6dd81cf None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:00.818136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b85a9-766c-4d98-bac7-7d8d35a94947 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.819147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b85a9-766c-4d98-bac7-7d8d35a94947 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.819841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b85a9-766c-4d98-bac7-7d8d35a94947 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.820797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b85a9-766c-4d98-bac7-7d8d35a94947 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.839988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b85a9-766c-4d98-bac7-7d8d35a94947 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.842865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b85a9-766c-4d98-bac7-7d8d35a94947 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.861587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 916/1826] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:00 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:00.877529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31e1b5e-d95e-4841-ae89-4154b6dd81cf None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:00.882325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7330a1e2-9094-4a01-88bf-0798d36641dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:00.883365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330a1e2-9094-4a01-88bf-0798d36641dd service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:00.883707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330a1e2-9094-4a01-88bf-0798d36641dd service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:00.884354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7330a1e2-9094-4a01-88bf-0798d36641dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:00.975898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7330a1e2-9094-4a01-88bf-0798d36641dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:00.979105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7330a1e2-9094-4a01-88bf-0798d36641dd service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:00.998582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 911/1827] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:00 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:01.484082 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b85a9-766c-4d98-bac7-7d8d35a94947 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:01.507855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:01.560653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b85a9-766c-4d98-bac7-7d8d35a94947 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:01.565205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:01.565875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:01.566168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:01.566543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] PATH_INFO: `/v3/users/e4464812190545f286efa68de4ba72c1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:01.568306 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4464812190545f286efa68de4ba72c1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:01.570341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:01.631300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4464812190545f286efa68de4ba72c1'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:01.632121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4464812190545f286efa68de4ba72c1'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:01.632895 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4464812190545f286efa68de4ba72c1'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:01.646099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4464812190545f286efa68de4ba72c1'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:01.659557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4464812190545f286efa68de4ba72c1'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:01.674632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 917/1828] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:01 2022] DELETE /identity/v3/users/e4464812190545f286efa68de4ba72c1 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:01.680123 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d847774-f6c2-47b0-a031-d14cffb42a00 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:01.680640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d847774-f6c2-47b0-a031-d14cffb42a00 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:01.680747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d847774-f6c2-47b0-a031-d14cffb42a00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:01.700453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6d847774-f6c2-47b0-a031-d14cffb42a00 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:01.701679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6d847774-f6c2-47b0-a031-d14cffb42a00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:01.726895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:01.765917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 912/1829] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:01.776318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:01.777819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:01.849116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92075c5-6a98-42d3-9d27-7afbefd86b8a admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:01.852877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc60efa0-1e8a-4296-a4b4-e9dfb2f2799e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:01.853703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc60efa0-1e8a-4296-a4b4-e9dfb2f2799e service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:01.853996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc60efa0-1e8a-4296-a4b4-e9dfb2f2799e service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:01.854278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc60efa0-1e8a-4296-a4b4-e9dfb2f2799e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:01.864103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc60efa0-1e8a-4296-a4b4-e9dfb2f2799e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:01.865621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc60efa0-1e8a-4296-a4b4-e9dfb2f2799e service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:01.875698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 918/1830] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:02.170792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d847774-f6c2-47b0-a031-d14cffb42a00 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:02.173306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6d847774-f6c2-47b0-a031-d14cffb42a00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:02.201680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d847774-f6c2-47b0-a031-d14cffb42a00 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:02.220375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:02.226625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:02.227469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:02.228228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] PATH_INFO: `/v3/projects/d86274d381754746bd01fbd52bb1ae57` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:02.233507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] RBAC: Authorizing `identity:delete_project(project_id=d86274d381754746bd01fbd52bb1ae57)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:02.240403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:02.352524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd86274d381754746bd01fbd52bb1ae57', 'request_id': 'req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:02.366405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd86274d381754746bd01fbd52bb1ae57', 'request_id': 'req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:02.400283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd86274d381754746bd01fbd52bb1ae57', 'request_id': 'req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:02.420457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] The token cache is being invalidate because project d86274d381754746bd01fbd52bb1ae57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:02.421918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:02.422841 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:02.426752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 913/1831] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:02 2022] DELETE /identity/v3/projects/d86274d381754746bd01fbd52bb1ae57 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:02.431004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc60efa0-1e8a-4296-a4b4-e9dfb2f2799e service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:02.447925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc60efa0-1e8a-4296-a4b4-e9dfb2f2799e service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:02.568743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:02.615217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:02.616145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:02.616900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:02.617770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] PATH_INFO: `/v3/users/2001f82027274d1dbf0280ea7d1610a6` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:02.619280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2001f82027274d1dbf0280ea7d1610a6)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:02.622355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:02.703213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2001f82027274d1dbf0280ea7d1610a6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:02.703886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2001f82027274d1dbf0280ea7d1610a6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:02.704516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2001f82027274d1dbf0280ea7d1610a6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:02.725109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2001f82027274d1dbf0280ea7d1610a6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:02.733414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-8901caa2-9929-46c5-ac97-382982f68959 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2001f82027274d1dbf0280ea7d1610a6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:02.754004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 919/1832] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:02 2022] DELETE /identity/v3/users/2001f82027274d1dbf0280ea7d1610a6 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:03.046550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:03.048161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:03.075296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d48189a-7922-4b8f-8fdb-5ac8ac2abcaf admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:03.139331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:03.139976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.140261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.140540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] PATH_INFO: `/v3/projects/6f8ee233f672493a95ce967e6e828fcf` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.141724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f8ee233f672493a95ce967e6e828fcf)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:03.143633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:03.188800 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f8ee233f672493a95ce967e6e828fcf', 'request_id': 'req-3ab4d229-2ce0-4be1-8415-079d2872f28d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:03.193131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:03.196969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f8ee233f672493a95ce967e6e828fcf', 'request_id': 'req-3ab4d229-2ce0-4be1-8415-079d2872f28d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:03.204502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f8ee233f672493a95ce967e6e828fcf', 'request_id': 'req-3ab4d229-2ce0-4be1-8415-079d2872f28d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:03.216282 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] The token cache is being invalidate because project 6f8ee233f672493a95ce967e6e828fcf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:03.216572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:03.217247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3ab4d229-2ce0-4be1-8415-079d2872f28d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:03.218754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 914/1833] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:03 2022] DELETE /identity/v3/projects/6f8ee233f672493a95ce967e6e828fcf => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:03.223737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb060d24-eeee-4ff4-9aeb-ad6007ed15a0 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.223987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb060d24-eeee-4ff4-9aeb-ad6007ed15a0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.224271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb060d24-eeee-4ff4-9aeb-ad6007ed15a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.243702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-eb060d24-eeee-4ff4-9aeb-ad6007ed15a0 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:03.244973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-eb060d24-eeee-4ff4-9aeb-ad6007ed15a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:03.303615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 920/1834] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:03.313770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ba9c6e-17d6-47b3-8383-022676e764d8 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.314113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ba9c6e-17d6-47b3-8383-022676e764d8 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.314394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ba9c6e-17d6-47b3-8383-022676e764d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.334396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-60ba9c6e-17d6-47b3-8383-022676e764d8 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:03.335602 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-60ba9c6e-17d6-47b3-8383-022676e764d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:03.343493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 915/1835] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:03.347702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb060d24-eeee-4ff4-9aeb-ad6007ed15a0 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:03.357408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb060d24-eeee-4ff4-9aeb-ad6007ed15a0 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:03.361959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d9212a-43df-465d-aae3-b13c0181ef79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:03.362571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d9212a-43df-465d-aae3-b13c0181ef79 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.362876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d9212a-43df-465d-aae3-b13c0181ef79 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.363729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d9212a-43df-465d-aae3-b13c0181ef79 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.364432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d9212a-43df-465d-aae3-b13c0181ef79 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:03.364776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d9212a-43df-465d-aae3-b13c0181ef79 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:03.366391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d9212a-43df-465d-aae3-b13c0181ef79 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:03.378016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 921/1836] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:48:03.382935 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ba9c6e-17d6-47b3-8383-022676e764d8 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:03.392327 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ba9c6e-17d6-47b3-8383-022676e764d8 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:03.397143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7240edbd-3fd6-45c7-a05b-85823a87af10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:03.397754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7240edbd-3fd6-45c7-a05b-85823a87af10 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.398042 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7240edbd-3fd6-45c7-a05b-85823a87af10 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.398325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7240edbd-3fd6-45c7-a05b-85823a87af10 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.398999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7240edbd-3fd6-45c7-a05b-85823a87af10 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:03.400942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7240edbd-3fd6-45c7-a05b-85823a87af10 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:03.429225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 916/1837] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:03 2022] POST /identity/v3/projects => generated 367 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:03.433523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d9212a-43df-465d-aae3-b13c0181ef79 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:03.444274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d9212a-43df-465d-aae3-b13c0181ef79 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:03.485472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:03.502747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9578bd9b-3ba5-4c92-a55d-4fed5280a429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:03.503371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9578bd9b-3ba5-4c92-a55d-4fed5280a429 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.503654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9578bd9b-3ba5-4c92-a55d-4fed5280a429 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.504778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9578bd9b-3ba5-4c92-a55d-4fed5280a429 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.505460 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9578bd9b-3ba5-4c92-a55d-4fed5280a429 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:03.507402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9578bd9b-3ba5-4c92-a55d-4fed5280a429 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:03.527688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 922/1838] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:03 2022] POST /identity/v3/users => generated 284 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:03.531740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7240edbd-3fd6-45c7-a05b-85823a87af10 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:03.542212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7240edbd-3fd6-45c7-a05b-85823a87af10 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:03.547317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d089dcd-fd70-4f69-bebe-e37fb1fd08ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:03.547931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d089dcd-fd70-4f69-bebe-e37fb1fd08ac admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.548212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d089dcd-fd70-4f69-bebe-e37fb1fd08ac admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.548487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d089dcd-fd70-4f69-bebe-e37fb1fd08ac admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.549313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d089dcd-fd70-4f69-bebe-e37fb1fd08ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:03.551565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d089dcd-fd70-4f69-bebe-e37fb1fd08ac admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:03.561823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 917/1839] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:03.565746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9578bd9b-3ba5-4c92-a55d-4fed5280a429 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:03.576994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9578bd9b-3ba5-4c92-a55d-4fed5280a429 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:03.581652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fa3e67-a21d-4baf-b998-69b392eabee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:03.582834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fa3e67-a21d-4baf-b998-69b392eabee4 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.583125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fa3e67-a21d-4baf-b998-69b392eabee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.584060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fa3e67-a21d-4baf-b998-69b392eabee4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.584813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fa3e67-a21d-4baf-b998-69b392eabee4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:03.586871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fa3e67-a21d-4baf-b998-69b392eabee4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:03.600319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 923/1840] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:03 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:03.611695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d089dcd-fd70-4f69-bebe-e37fb1fd08ac admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:03.622630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d089dcd-fd70-4f69-bebe-e37fb1fd08ac admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:03.627318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b90922-fb5e-4adf-8381-daebf5708bf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:03.628057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b90922-fb5e-4adf-8381-daebf5708bf2 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.628386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b90922-fb5e-4adf-8381-daebf5708bf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.628711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b90922-fb5e-4adf-8381-daebf5708bf2 admin admin] PATH_INFO: `/v3/projects/d5f63577dcb2496bbefadc44b28beccb/users/779518b2969242268bae0c968c204959/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.644351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b90922-fb5e-4adf-8381-daebf5708bf2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5f63577dcb2496bbefadc44b28beccb, user_id=779518b2969242268bae0c968c204959, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:03.647279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b90922-fb5e-4adf-8381-daebf5708bf2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:03.669810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 918/1841] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:03 2022] PUT /identity/v3/projects/d5f63577dcb2496bbefadc44b28beccb/users/779518b2969242268bae0c968c204959/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:03.681512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979dda72-8435-46b5-8a6c-180d8e542256 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.681809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979dda72-8435-46b5-8a6c-180d8e542256 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.682084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979dda72-8435-46b5-8a6c-180d8e542256 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.713061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-979dda72-8435-46b5-8a6c-180d8e542256 None None] MFA Rules not processed for user `779518b2969242268bae0c968c204959`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:03.714291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-979dda72-8435-46b5-8a6c-180d8e542256 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:03.792244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 924/1842] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:03 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:03.797184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b90922-fb5e-4adf-8381-daebf5708bf2 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:03.861836 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b90922-fb5e-4adf-8381-daebf5708bf2 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:03.868263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e119d8-de44-4d79-a759-4ead1b72ad69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:03.875293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e119d8-de44-4d79-a759-4ead1b72ad69 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.875601 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e119d8-de44-4d79-a759-4ead1b72ad69 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.875891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e119d8-de44-4d79-a759-4ead1b72ad69 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.876591 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e119d8-de44-4d79-a759-4ead1b72ad69 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:03.878885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e119d8-de44-4d79-a759-4ead1b72ad69 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:03.909832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 919/1843] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:03 2022] POST /identity/v3/projects => generated 367 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:03.911940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979dda72-8435-46b5-8a6c-180d8e542256 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:03.923802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979dda72-8435-46b5-8a6c-180d8e542256 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:03.987756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969d1964-eb5c-4e47-8edd-0ed28efb7058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:03.989348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969d1964-eb5c-4e47-8edd-0ed28efb7058 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:03.989845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969d1964-eb5c-4e47-8edd-0ed28efb7058 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:03.990128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969d1964-eb5c-4e47-8edd-0ed28efb7058 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:03.990815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969d1964-eb5c-4e47-8edd-0ed28efb7058 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:03.992929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969d1964-eb5c-4e47-8edd-0ed28efb7058 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:04.016709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 925/1844] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:03 2022] POST /identity/v3/users => generated 290 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:04.018934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e119d8-de44-4d79-a759-4ead1b72ad69 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:04.029898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e119d8-de44-4d79-a759-4ead1b72ad69 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:04.034626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b029b53e-c650-4eb6-a6a4-92328caef496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:04.035946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b029b53e-c650-4eb6-a6a4-92328caef496 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:04.036324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b029b53e-c650-4eb6-a6a4-92328caef496 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:04.036628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b029b53e-c650-4eb6-a6a4-92328caef496 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:04.037335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b029b53e-c650-4eb6-a6a4-92328caef496 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:04.038949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b029b53e-c650-4eb6-a6a4-92328caef496 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:04.050022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 920/1845] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:04.054838 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969d1964-eb5c-4e47-8edd-0ed28efb7058 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:04.064773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969d1964-eb5c-4e47-8edd-0ed28efb7058 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:04.069429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0139337-50a5-4f7b-bd1f-360c9dba07af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:04.070166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0139337-50a5-4f7b-bd1f-360c9dba07af admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:04.070452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0139337-50a5-4f7b-bd1f-360c9dba07af admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:04.070731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0139337-50a5-4f7b-bd1f-360c9dba07af admin admin] PATH_INFO: `/v3/projects/66d9f7643ec84f7abc878274eac14a63/users/36cde2a609da49a88e8192949df7fef6/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:04.086029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0139337-50a5-4f7b-bd1f-360c9dba07af admin admin] RBAC: Authorizing `identity:create_grant(project_id=66d9f7643ec84f7abc878274eac14a63, user_id=36cde2a609da49a88e8192949df7fef6, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:04.088764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0139337-50a5-4f7b-bd1f-360c9dba07af admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:04.102227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 926/1846] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:04 2022] PUT /identity/v3/projects/66d9f7643ec84f7abc878274eac14a63/users/36cde2a609da49a88e8192949df7fef6/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:04.106110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b029b53e-c650-4eb6-a6a4-92328caef496 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:04.117334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b029b53e-c650-4eb6-a6a4-92328caef496 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:04.166085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:04.177640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dea5636-73a8-4ce7-beac-c3434b0f985c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:04.178277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dea5636-73a8-4ce7-beac-c3434b0f985c admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:04.178568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dea5636-73a8-4ce7-beac-c3434b0f985c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:04.178847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dea5636-73a8-4ce7-beac-c3434b0f985c admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:04.179585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dea5636-73a8-4ce7-beac-c3434b0f985c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:04.181251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dea5636-73a8-4ce7-beac-c3434b0f985c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:04.191817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 921/1847] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:04 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:04.195769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0139337-50a5-4f7b-bd1f-360c9dba07af admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:04.206024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0139337-50a5-4f7b-bd1f-360c9dba07af admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:04.211457 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec9d226-d59d-4bec-b406-8c5a364d5251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:04.212132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec9d226-d59d-4bec-b406-8c5a364d5251 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:04.212421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec9d226-d59d-4bec-b406-8c5a364d5251 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:04.212865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec9d226-d59d-4bec-b406-8c5a364d5251 admin admin] PATH_INFO: `/v3/projects/66d9f7643ec84f7abc878274eac14a63/users/36cde2a609da49a88e8192949df7fef6/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:04.214852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec9d226-d59d-4bec-b406-8c5a364d5251 admin admin] RBAC: Authorizing `identity:create_grant(project_id=66d9f7643ec84f7abc878274eac14a63, user_id=36cde2a609da49a88e8192949df7fef6, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:04.217528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec9d226-d59d-4bec-b406-8c5a364d5251 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:04.229656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 927/1848] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:04 2022] PUT /identity/v3/projects/66d9f7643ec84f7abc878274eac14a63/users/36cde2a609da49a88e8192949df7fef6/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:04.239933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f01abec-c237-4c58-b203-987bb7412c3a None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:04.240241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f01abec-c237-4c58-b203-987bb7412c3a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:04.240530 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f01abec-c237-4c58-b203-987bb7412c3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:04.269191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-8f01abec-c237-4c58-b203-987bb7412c3a None None] MFA Rules not processed for user `36cde2a609da49a88e8192949df7fef6`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:04.271076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8f01abec-c237-4c58-b203-987bb7412c3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:04.338481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 922/1849] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:04 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:04.350198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec9d226-d59d-4bec-b406-8c5a364d5251 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:04.351873 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-6ec9d226-d59d-4bec-b406-8c5a364d5251 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:04.358355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:04.409609 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec9d226-d59d-4bec-b406-8c5a364d5251 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:04.414362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bc81e3-60a6-4f92-ab60-36da4d25ffae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:04.415017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc81e3-60a6-4f92-ab60-36da4d25ffae service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:04.415308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc81e3-60a6-4f92-ab60-36da4d25ffae service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:04.415590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bc81e3-60a6-4f92-ab60-36da4d25ffae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:04.424975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bc81e3-60a6-4f92-ab60-36da4d25ffae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:04.426468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bc81e3-60a6-4f92-ab60-36da4d25ffae service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:04.435843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 928/1850] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:04 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:05.843110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f01abec-c237-4c58-b203-987bb7412c3a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:05.905186 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f01abec-c237-4c58-b203-987bb7412c3a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:05.909662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:05.910373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:05.910627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:05.910901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] PATH_INFO: `/v3/users/779518b2969242268bae0c968c204959` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:05.912062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] RBAC: Authorizing `identity:delete_user(user_id=779518b2969242268bae0c968c204959)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:05.914006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:05.961210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '779518b2969242268bae0c968c204959'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:05.969032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '779518b2969242268bae0c968c204959'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:05.969667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '779518b2969242268bae0c968c204959'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:05.970292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '779518b2969242268bae0c968c204959'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:05.977246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '779518b2969242268bae0c968c204959'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:05.987276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:06.001110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 923/1851] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:05 2022] DELETE /identity/v3/users/779518b2969242268bae0c968c204959 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:06.004521 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13348f6-55e0-465d-99a2-b5271c332a46 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:06.005034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13348f6-55e0-465d-99a2-b5271c332a46 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:06.005394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13348f6-55e0-465d-99a2-b5271c332a46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:06.025833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a13348f6-55e0-465d-99a2-b5271c332a46 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:06.027069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a13348f6-55e0-465d-99a2-b5271c332a46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:06.097097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 929/1852] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:06.107422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:06.109615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:06.197578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f44998c-6d58-4527-a6fb-56870db7273f admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:06.201801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b5049d-06f0-4cdd-83e4-43e1952316c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:06.202625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5049d-06f0-4cdd-83e4-43e1952316c7 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:06.202947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5049d-06f0-4cdd-83e4-43e1952316c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:06.203307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5049d-06f0-4cdd-83e4-43e1952316c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:06.213946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b5049d-06f0-4cdd-83e4-43e1952316c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:06.215407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b5049d-06f0-4cdd-83e4-43e1952316c7 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:06.226293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 924/1853] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:06.530230 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13348f6-55e0-465d-99a2-b5271c332a46 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:06.532035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a13348f6-55e0-465d-99a2-b5271c332a46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:06.545559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13348f6-55e0-465d-99a2-b5271c332a46 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:06.550879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:06.551723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:06.552109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:06.552487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] PATH_INFO: `/v3/projects/d5f63577dcb2496bbefadc44b28beccb` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:06.553946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d5f63577dcb2496bbefadc44b28beccb)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:06.555913 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:06.607466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd5f63577dcb2496bbefadc44b28beccb', 'request_id': 'req-b73e665c-7121-4369-b773-d333a8ae6b9c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:06.613672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd5f63577dcb2496bbefadc44b28beccb', 'request_id': 'req-b73e665c-7121-4369-b773-d333a8ae6b9c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:06.621347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd5f63577dcb2496bbefadc44b28beccb', 'request_id': 'req-b73e665c-7121-4369-b773-d333a8ae6b9c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:06.636763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] The token cache is being invalidate because project d5f63577dcb2496bbefadc44b28beccb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:06.637057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:06.637706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:06.639222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 930/1854] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:06 2022] DELETE /identity/v3/projects/d5f63577dcb2496bbefadc44b28beccb => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:06.643087 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b5049d-06f0-4cdd-83e4-43e1952316c7 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:06.653747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b5049d-06f0-4cdd-83e4-43e1952316c7 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:06.710740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:06.711389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:06.711669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:06.711946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] PATH_INFO: `/v3/users/36cde2a609da49a88e8192949df7fef6` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:06.713128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] RBAC: Authorizing `identity:delete_user(user_id=36cde2a609da49a88e8192949df7fef6)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:06.715143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:06.764361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '36cde2a609da49a88e8192949df7fef6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:06.773368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '36cde2a609da49a88e8192949df7fef6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:06.774020 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '36cde2a609da49a88e8192949df7fef6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:06.774669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '36cde2a609da49a88e8192949df7fef6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:06.783113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '36cde2a609da49a88e8192949df7fef6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:06.801265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 925/1855] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:06 2022] DELETE /identity/v3/users/36cde2a609da49a88e8192949df7fef6 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:07.078405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:07.084099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:07.109652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73e665c-7121-4369-b773-d333a8ae6b9c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:07.143274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:07.158007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:07.159538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:07.179027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:07.183481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:07.184226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.184523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.184847 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] PATH_INFO: `/v3/projects/66d9f7643ec84f7abc878274eac14a63` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.186475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] RBAC: Authorizing `identity:delete_project(project_id=66d9f7643ec84f7abc878274eac14a63)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:07.188917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:07.234572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcd8c9a-8972-41ad-85f3-9f9e4bc17d97 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:07.245475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66d9f7643ec84f7abc878274eac14a63', 'request_id': 'req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:07.253495 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66d9f7643ec84f7abc878274eac14a63', 'request_id': 'req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:07.263829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66d9f7643ec84f7abc878274eac14a63', 'request_id': 'req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:07.277343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] The token cache is being invalidate because project 66d9f7643ec84f7abc878274eac14a63 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:07.277733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:07.278490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3e3e1915-77e3-4aa1-8ee5-eb12d23da39a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:07.280542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 931/1856] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:07 2022] DELETE /identity/v3/projects/66d9f7643ec84f7abc878274eac14a63 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:07.285051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f451ac-ea31-417c-86ed-6dd49fc3347c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.285338 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f451ac-ea31-417c-86ed-6dd49fc3347c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.285620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f451ac-ea31-417c-86ed-6dd49fc3347c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.297382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:07.298026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.298317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.298677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] PATH_INFO: `/v3/users/f50ab8be666243708e1ad10889cdaba3` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.299967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] RBAC: Authorizing `identity:delete_user(user_id=f50ab8be666243708e1ad10889cdaba3)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:07.302160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:07.308846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-16f451ac-ea31-417c-86ed-6dd49fc3347c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:07.309841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-16f451ac-ea31-417c-86ed-6dd49fc3347c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:07.323151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 932/1857] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:07.333080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebf5186-e47b-4226-94e4-8438d571c578 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.333367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebf5186-e47b-4226-94e4-8438d571c578 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.334095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebf5186-e47b-4226-94e4-8438d571c578 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.365085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f50ab8be666243708e1ad10889cdaba3'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:07.373249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-aebf5186-e47b-4226-94e4-8438d571c578 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:07.374485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-aebf5186-e47b-4226-94e4-8438d571c578 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:07.376671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f50ab8be666243708e1ad10889cdaba3'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:07.377279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f50ab8be666243708e1ad10889cdaba3'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:07.377891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f50ab8be666243708e1ad10889cdaba3'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:07.384046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 933/1858] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:07.386457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-59b0e3ed-d78f-403c-af1e-bfc5fc04400f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f50ab8be666243708e1ad10889cdaba3'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:07.388058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebf5186-e47b-4226-94e4-8438d571c578 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:07.399565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebf5186-e47b-4226-94e4-8438d571c578 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:07.406170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bf21ed-c991-4cc3-a5b9-7816e09c521e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:07.406818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bf21ed-c991-4cc3-a5b9-7816e09c521e admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.407388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bf21ed-c991-4cc3-a5b9-7816e09c521e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.407678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bf21ed-c991-4cc3-a5b9-7816e09c521e admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.408398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bf21ed-c991-4cc3-a5b9-7816e09c521e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:07.408812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bf21ed-c991-4cc3-a5b9-7816e09c521e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:07.410473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bf21ed-c991-4cc3-a5b9-7816e09c521e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:07.411577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 926/1859] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:07 2022] DELETE /identity/v3/users/f50ab8be666243708e1ad10889cdaba3 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:07.416866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51ff8ee-dd3b-4ede-98ca-8f36c59affc1 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.417157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51ff8ee-dd3b-4ede-98ca-8f36c59affc1 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.417454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51ff8ee-dd3b-4ede-98ca-8f36c59affc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.430969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 934/1860] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:48:07.435761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bf21ed-c991-4cc3-a5b9-7816e09c521e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:07.452100 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bf21ed-c991-4cc3-a5b9-7816e09c521e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:07.453825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-b51ff8ee-dd3b-4ede-98ca-8f36c59affc1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:07.455174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b51ff8ee-dd3b-4ede-98ca-8f36c59affc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:07.528159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4304e-a933-433f-8fd6-60195c56546f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:07.528978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4304e-a933-433f-8fd6-60195c56546f admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.529279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4304e-a933-433f-8fd6-60195c56546f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.529563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd4304e-a933-433f-8fd6-60195c56546f admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.530244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd4304e-a933-433f-8fd6-60195c56546f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:07.532176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd4304e-a933-433f-8fd6-60195c56546f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:07.540852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:07.566715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 935/1861] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:07 2022] POST /identity/v3/projects => generated 401 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:07.571079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4304e-a933-433f-8fd6-60195c56546f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:07.572306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 927/1862] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:07.583765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51ff8ee-dd3b-4ede-98ca-8f36c59affc1 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:07.585245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b51ff8ee-dd3b-4ede-98ca-8f36c59affc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:07.594670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:07.644847 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd4304e-a933-433f-8fd6-60195c56546f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:07.652798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a305a762-b95a-4ef3-a2e4-499cb33388df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:07.653447 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a305a762-b95a-4ef3-a2e4-499cb33388df admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.653752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a305a762-b95a-4ef3-a2e4-499cb33388df admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.654007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a305a762-b95a-4ef3-a2e4-499cb33388df admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.654687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a305a762-b95a-4ef3-a2e4-499cb33388df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:07.655880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51ff8ee-dd3b-4ede-98ca-8f36c59affc1 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:07.656744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a305a762-b95a-4ef3-a2e4-499cb33388df admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:07.684779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 936/1863] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:07 2022] POST /identity/v3/users => generated 301 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:07.686374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a305a762-b95a-4ef3-a2e4-499cb33388df admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:07.697157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a305a762-b95a-4ef3-a2e4-499cb33388df admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:07.702224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fc44d8-c874-411c-830b-8538e36fe5a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:07.702855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc44d8-c874-411c-830b-8538e36fe5a3 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.703143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc44d8-c874-411c-830b-8538e36fe5a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.703423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fc44d8-c874-411c-830b-8538e36fe5a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.704131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fc44d8-c874-411c-830b-8538e36fe5a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:07.705985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fc44d8-c874-411c-830b-8538e36fe5a3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:07.715314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6850c4e0-d65d-436d-87aa-092519ff1998 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:07.715950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6850c4e0-d65d-436d-87aa-092519ff1998 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.716236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6850c4e0-d65d-436d-87aa-092519ff1998 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.716533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6850c4e0-d65d-436d-87aa-092519ff1998 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.719881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 937/1864] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:07 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:07.720803 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fc44d8-c874-411c-830b-8538e36fe5a3 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:07.729309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6850c4e0-d65d-436d-87aa-092519ff1998 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:07.730662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fc44d8-c874-411c-830b-8538e36fe5a3 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:07.732135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6850c4e0-d65d-436d-87aa-092519ff1998 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:07.735553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813a1919-78a2-4656-85d0-ccfd128bdf23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:07.736157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813a1919-78a2-4656-85d0-ccfd128bdf23 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.736449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813a1919-78a2-4656-85d0-ccfd128bdf23 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.736776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813a1919-78a2-4656-85d0-ccfd128bdf23 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.737470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813a1919-78a2-4656-85d0-ccfd128bdf23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:07.739614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813a1919-78a2-4656-85d0-ccfd128bdf23 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:07.749589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 928/1865] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:07.750648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 938/1866] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:07.755156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6850c4e0-d65d-436d-87aa-092519ff1998 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:07.764862 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6850c4e0-d65d-436d-87aa-092519ff1998 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:07.769689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72eb1235-448b-447b-8f36-69d534cc237b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:07.770363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eb1235-448b-447b-8f36-69d534cc237b admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.770766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eb1235-448b-447b-8f36-69d534cc237b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.771053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72eb1235-448b-447b-8f36-69d534cc237b admin admin] PATH_INFO: `/v3/projects/c38ef094040b413cb1a7c37a4202c758/users/979b92659a6d4615a8df21688ef6e420/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.785653 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72eb1235-448b-447b-8f36-69d534cc237b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c38ef094040b413cb1a7c37a4202c758, user_id=979b92659a6d4615a8df21688ef6e420, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:07.788290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72eb1235-448b-447b-8f36-69d534cc237b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:07.816729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 929/1867] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:07 2022] PUT /identity/v3/projects/c38ef094040b413cb1a7c37a4202c758/users/979b92659a6d4615a8df21688ef6e420/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:07.822675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac01bb-8990-473f-bca1-de08cbb56d47 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:07.823132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac01bb-8990-473f-bca1-de08cbb56d47 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:07.823350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac01bb-8990-473f-bca1-de08cbb56d47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:07.866282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-94ac01bb-8990-473f-bca1-de08cbb56d47 None None] MFA Rules not processed for user `979b92659a6d4615a8df21688ef6e420`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:07.869381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-94ac01bb-8990-473f-bca1-de08cbb56d47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:07.940044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72eb1235-448b-447b-8f36-69d534cc237b admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:07.942499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-72eb1235-448b-447b-8f36-69d534cc237b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:08.002681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 939/1868] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:07 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:08.007380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac01bb-8990-473f-bca1-de08cbb56d47 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:08.029674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72eb1235-448b-447b-8f36-69d534cc237b admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:08.038483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09b7bde-46d5-45e3-8c8a-09e6d23e79ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:08.039182 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09b7bde-46d5-45e3-8c8a-09e6d23e79ed service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.039478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09b7bde-46d5-45e3-8c8a-09e6d23e79ed service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.040292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09b7bde-46d5-45e3-8c8a-09e6d23e79ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.042595 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d09b7bde-46d5-45e3-8c8a-09e6d23e79ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:08.109496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac01bb-8990-473f-bca1-de08cbb56d47 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:08.114361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:08.115803 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16d290a-5782-4ab0-aad9-2e9642b22218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:08.116785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d290a-5782-4ab0-aad9-2e9642b22218 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.117105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d290a-5782-4ab0-aad9-2e9642b22218 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.117473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a16d290a-5782-4ab0-aad9-2e9642b22218 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.118229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16d290a-5782-4ab0-aad9-2e9642b22218 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:08.120304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a16d290a-5782-4ab0-aad9-2e9642b22218 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:08.160434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 940/1869] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:08 2022] POST /identity/v3/projects => generated 401 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:08.161649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16d290a-5782-4ab0-aad9-2e9642b22218 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:08.168711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:08.177270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09b7bde-46d5-45e3-8c8a-09e6d23e79ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:08.178914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09b7bde-46d5-45e3-8c8a-09e6d23e79ed service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:08.202275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 930/1870] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:07 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:08.203806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09b7bde-46d5-45e3-8c8a-09e6d23e79ed service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:08.217075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d09b7bde-46d5-45e3-8c8a-09e6d23e79ed service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:08.241010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09b7bde-46d5-45e3-8c8a-09e6d23e79ed service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:08.255696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a16d290a-5782-4ab0-aad9-2e9642b22218 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:08.262473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e483c8f-698d-4613-9004-7c17dfd6bcd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:08.263652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e483c8f-698d-4613-9004-7c17dfd6bcd8 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.264152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e483c8f-698d-4613-9004-7c17dfd6bcd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.264630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e483c8f-698d-4613-9004-7c17dfd6bcd8 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.266055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e483c8f-698d-4613-9004-7c17dfd6bcd8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:08.268576 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e483c8f-698d-4613-9004-7c17dfd6bcd8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:08.301563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 941/1871] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:08 2022] POST /identity/v3/users => generated 307 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:08.302645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e483c8f-698d-4613-9004-7c17dfd6bcd8 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:08.316431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e483c8f-698d-4613-9004-7c17dfd6bcd8 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:08.321577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8202679-8095-4bc6-a0c2-4e944e6b6d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:08.322266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8202679-8095-4bc6-a0c2-4e944e6b6d92 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.322518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8202679-8095-4bc6-a0c2-4e944e6b6d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.322776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8202679-8095-4bc6-a0c2-4e944e6b6d92 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.323481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8202679-8095-4bc6-a0c2-4e944e6b6d92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:08.325140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8202679-8095-4bc6-a0c2-4e944e6b6d92 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:08.325527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:08.326183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.326478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.326756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] PATH_INFO: `/v3/projects/a49ff0f2869741fb914ec3fbcd786b1f` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.328356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a49ff0f2869741fb914ec3fbcd786b1f)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:08.331219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:08.339343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 942/1872] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:08.340700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8202679-8095-4bc6-a0c2-4e944e6b6d92 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:08.352756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8202679-8095-4bc6-a0c2-4e944e6b6d92 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:08.358489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432a47a1-2560-4a2a-bfcc-5181a6f709c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:08.359319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a47a1-2560-4a2a-bfcc-5181a6f709c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.359757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a47a1-2560-4a2a-bfcc-5181a6f709c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.360015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a47a1-2560-4a2a-bfcc-5181a6f709c6 admin admin] PATH_INFO: `/v3/projects/1995f18257274b8ab36d7e390a8f7e3c/users/04a8f3c55a5748768a65f65f82210198/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.384843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432a47a1-2560-4a2a-bfcc-5181a6f709c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1995f18257274b8ab36d7e390a8f7e3c, user_id=04a8f3c55a5748768a65f65f82210198, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:08.387584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432a47a1-2560-4a2a-bfcc-5181a6f709c6 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:08.393377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a49ff0f2869741fb914ec3fbcd786b1f', 'request_id': 'req-db365c8e-4cd2-4698-a59d-714315290e69'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:08.409586 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a49ff0f2869741fb914ec3fbcd786b1f', 'request_id': 'req-db365c8e-4cd2-4698-a59d-714315290e69'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:08.411625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 943/1873] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:08 2022] PUT /identity/v3/projects/1995f18257274b8ab36d7e390a8f7e3c/users/04a8f3c55a5748768a65f65f82210198/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:08.415849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432a47a1-2560-4a2a-bfcc-5181a6f709c6 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:08.416730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a49ff0f2869741fb914ec3fbcd786b1f', 'request_id': 'req-db365c8e-4cd2-4698-a59d-714315290e69'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:08.431374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] The token cache is being invalidate because project a49ff0f2869741fb914ec3fbcd786b1f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:08.431675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:08.432394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:08.434185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 931/1874] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:08 2022] DELETE /identity/v3/projects/a49ff0f2869741fb914ec3fbcd786b1f => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:08.438040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:08.439297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:08.502017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432a47a1-2560-4a2a-bfcc-5181a6f709c6 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:08.509044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a197808-4390-48c7-b5c6-e5d29e9da0aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:08.509881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a197808-4390-48c7-b5c6-e5d29e9da0aa admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.510923 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a197808-4390-48c7-b5c6-e5d29e9da0aa admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.511218 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a197808-4390-48c7-b5c6-e5d29e9da0aa admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.511939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a197808-4390-48c7-b5c6-e5d29e9da0aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:08.514591 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:08.514900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a197808-4390-48c7-b5c6-e5d29e9da0aa admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:08.520695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db365c8e-4cd2-4698-a59d-714315290e69 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:08.527821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:08.530104 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.530363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 944/1875] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:08 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:08.530454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.530676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] PATH_INFO: `/v3/users/707564ea64354507a0efb1aee1801de4` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.533606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=707564ea64354507a0efb1aee1801de4)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:08.536419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a197808-4390-48c7-b5c6-e5d29e9da0aa admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:08.538470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:08.538775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-1a197808-4390-48c7-b5c6-e5d29e9da0aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:08.576508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a197808-4390-48c7-b5c6-e5d29e9da0aa admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:08.589165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7d4a8f-b120-45d4-9c63-e1846016fa35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:08.590430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7d4a8f-b120-45d4-9c63-e1846016fa35 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.590851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7d4a8f-b120-45d4-9c63-e1846016fa35 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.591275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7d4a8f-b120-45d4-9c63-e1846016fa35 admin admin] PATH_INFO: `/v3/projects/1995f18257274b8ab36d7e390a8f7e3c/users/04a8f3c55a5748768a65f65f82210198/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.595060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7d4a8f-b120-45d4-9c63-e1846016fa35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1995f18257274b8ab36d7e390a8f7e3c, user_id=04a8f3c55a5748768a65f65f82210198, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:08.602956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7d4a8f-b120-45d4-9c63-e1846016fa35 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:08.626229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 945/1876] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:08 2022] PUT /identity/v3/projects/1995f18257274b8ab36d7e390a8f7e3c/users/04a8f3c55a5748768a65f65f82210198/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:08.637701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff1cf9-9d84-4041-806e-8a7dd44447de None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.638028 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff1cf9-9d84-4041-806e-8a7dd44447de None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.638345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbff1cf9-9d84-4041-806e-8a7dd44447de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.653880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '707564ea64354507a0efb1aee1801de4'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:08.666402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '707564ea64354507a0efb1aee1801de4'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:08.666402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '707564ea64354507a0efb1aee1801de4'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:08.666402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '707564ea64354507a0efb1aee1801de4'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:08.676290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '707564ea64354507a0efb1aee1801de4'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:08.700284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 932/1877] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:08 2022] DELETE /identity/v3/users/707564ea64354507a0efb1aee1801de4 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:08.706336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-fbff1cf9-9d84-4041-806e-8a7dd44447de None None] MFA Rules not processed for user `04a8f3c55a5748768a65f65f82210198`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:08.709053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fbff1cf9-9d84-4041-806e-8a7dd44447de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:08.746192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:08.807989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 946/1878] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:08 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:08.821359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:08.823860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:08.933621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc59f398-c4a0-4e17-984a-adcb706987f0 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:08.938313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4e6821-8304-4aa5-929a-7b2cbfb63dcd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:08.938940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4e6821-8304-4aa5-929a-7b2cbfb63dcd service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:08.939225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4e6821-8304-4aa5-929a-7b2cbfb63dcd service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:08.939502 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4e6821-8304-4aa5-929a-7b2cbfb63dcd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:08.948990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4e6821-8304-4aa5-929a-7b2cbfb63dcd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:08.950429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4e6821-8304-4aa5-929a-7b2cbfb63dcd service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:08.962184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 933/1879] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:08 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:09.057944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff1cf9-9d84-4041-806e-8a7dd44447de None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:09.071935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbff1cf9-9d84-4041-806e-8a7dd44447de None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:09.079802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1aa038a-6fb1-407d-929d-6c554c68e76c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:09.080447 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aa038a-6fb1-407d-929d-6c554c68e76c service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.080943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aa038a-6fb1-407d-929d-6c554c68e76c service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.081538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1aa038a-6fb1-407d-929d-6c554c68e76c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.083706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d1aa038a-6fb1-407d-929d-6c554c68e76c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:09.132818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4e6821-8304-4aa5-929a-7b2cbfb63dcd service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:09.134448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-1d4e6821-8304-4aa5-929a-7b2cbfb63dcd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:09.150748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1aa038a-6fb1-407d-929d-6c554c68e76c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:09.152213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1aa038a-6fb1-407d-929d-6c554c68e76c service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:09.165416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 947/1880] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:09 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:09.208694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4e6821-8304-4aa5-929a-7b2cbfb63dcd service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:09.214742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:09.215778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.216058 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.216421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] PATH_INFO: `/v3/projects/6673e4bdeffb40aca77e2740b415c15b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.217875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6673e4bdeffb40aca77e2740b415c15b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:09.219839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:09.247828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:09.276672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6673e4bdeffb40aca77e2740b415c15b', 'request_id': 'req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:09.285868 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6673e4bdeffb40aca77e2740b415c15b', 'request_id': 'req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:09.292632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6673e4bdeffb40aca77e2740b415c15b', 'request_id': 'req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:09.309416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] The token cache is being invalidate because project 6673e4bdeffb40aca77e2740b415c15b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:09.309767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:09.310790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d9b16e5c-62c7-4a98-acde-bf55e06a3ab2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:09.312838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 934/1881] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:09 2022] DELETE /identity/v3/projects/6673e4bdeffb40aca77e2740b415c15b => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:09.318715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0aa346-2482-4e4f-ae4f-1e6581cb6390 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.319015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0aa346-2482-4e4f-ae4f-1e6581cb6390 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.319303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0aa346-2482-4e4f-ae4f-1e6581cb6390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.345163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-0a0aa346-2482-4e4f-ae4f-1e6581cb6390 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:09.346540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0a0aa346-2482-4e4f-ae4f-1e6581cb6390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:09.434416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 948/1882] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:09.443079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed767b51-0577-4fcb-af89-87f4594f963d None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.443351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed767b51-0577-4fcb-af89-87f4594f963d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.443633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed767b51-0577-4fcb-af89-87f4594f963d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.479336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-ed767b51-0577-4fcb-af89-87f4594f963d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:09.480593 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ed767b51-0577-4fcb-af89-87f4594f963d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:09.490882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 935/1883] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:09.494977 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0aa346-2482-4e4f-ae4f-1e6581cb6390 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:09.506166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0aa346-2482-4e4f-ae4f-1e6581cb6390 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:09.516313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42613176-d76b-40cc-9eb6-a5b4d8560772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:09.517332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42613176-d76b-40cc-9eb6-a5b4d8560772 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.517889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42613176-d76b-40cc-9eb6-a5b4d8560772 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.518452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42613176-d76b-40cc-9eb6-a5b4d8560772 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.519450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42613176-d76b-40cc-9eb6-a5b4d8560772 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:09.520227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42613176-d76b-40cc-9eb6-a5b4d8560772 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:09.522214 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42613176-d76b-40cc-9eb6-a5b4d8560772 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:09.544213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 949/1884] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:48:09.546333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed767b51-0577-4fcb-af89-87f4594f963d None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:09.558063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed767b51-0577-4fcb-af89-87f4594f963d None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:09.563375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc64419-25f4-4fb9-b451-2e8e3539704a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:09.564090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc64419-25f4-4fb9-b451-2e8e3539704a admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.564565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc64419-25f4-4fb9-b451-2e8e3539704a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.564848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc64419-25f4-4fb9-b451-2e8e3539704a admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.565607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc64419-25f4-4fb9-b451-2e8e3539704a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:09.568017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc64419-25f4-4fb9-b451-2e8e3539704a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:09.599843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 936/1885] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:09 2022] POST /identity/v3/projects => generated 381 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:09.604383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42613176-d76b-40cc-9eb6-a5b4d8560772 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:09.617317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42613176-d76b-40cc-9eb6-a5b4d8560772 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:09.684254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:09.696403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e7f4fc-b3b4-43b4-b130-82852a6c0c7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:09.697619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e7f4fc-b3b4-43b4-b130-82852a6c0c7b admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.697925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e7f4fc-b3b4-43b4-b130-82852a6c0c7b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.698209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e7f4fc-b3b4-43b4-b130-82852a6c0c7b admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.699191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e7f4fc-b3b4-43b4-b130-82852a6c0c7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:09.701184 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e7f4fc-b3b4-43b4-b130-82852a6c0c7b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:09.730384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 950/1886] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:09 2022] POST /identity/v3/users => generated 291 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:09.738822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc64419-25f4-4fb9-b451-2e8e3539704a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:09.750420 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc64419-25f4-4fb9-b451-2e8e3539704a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:09.755376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16afa4f5-ada8-4299-b73f-7db3e91c4cb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:09.756004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16afa4f5-ada8-4299-b73f-7db3e91c4cb8 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.756290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16afa4f5-ada8-4299-b73f-7db3e91c4cb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.756570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16afa4f5-ada8-4299-b73f-7db3e91c4cb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.757309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16afa4f5-ada8-4299-b73f-7db3e91c4cb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:09.758985 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16afa4f5-ada8-4299-b73f-7db3e91c4cb8 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:09.772634 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 937/1887] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:09.777446 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e7f4fc-b3b4-43b4-b130-82852a6c0c7b admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:09.788756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e7f4fc-b3b4-43b4-b130-82852a6c0c7b admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:09.793422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20744cff-1165-4323-bad7-92ff8f08de24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:09.794041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20744cff-1165-4323-bad7-92ff8f08de24 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.794332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20744cff-1165-4323-bad7-92ff8f08de24 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.794612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20744cff-1165-4323-bad7-92ff8f08de24 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.795310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20744cff-1165-4323-bad7-92ff8f08de24 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:09.797199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20744cff-1165-4323-bad7-92ff8f08de24 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:09.811952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 951/1888] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:09.816095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16afa4f5-ada8-4299-b73f-7db3e91c4cb8 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:09.827236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16afa4f5-ada8-4299-b73f-7db3e91c4cb8 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:09.833065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3d6549-91bc-45c7-ac9a-31b88ac402ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:09.833751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3d6549-91bc-45c7-ac9a-31b88ac402ac admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.834040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3d6549-91bc-45c7-ac9a-31b88ac402ac admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.834322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3d6549-91bc-45c7-ac9a-31b88ac402ac admin admin] PATH_INFO: `/v3/projects/6dde749e999748eca35eda09181f0766/users/b9a543a8b4f94fa7bdcc0a3b7c52396e/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.850066 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3d6549-91bc-45c7-ac9a-31b88ac402ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=6dde749e999748eca35eda09181f0766, user_id=b9a543a8b4f94fa7bdcc0a3b7c52396e, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:09.852780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3d6549-91bc-45c7-ac9a-31b88ac402ac admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:09.866366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 938/1889] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:09 2022] PUT /identity/v3/projects/6dde749e999748eca35eda09181f0766/users/b9a543a8b4f94fa7bdcc0a3b7c52396e/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:09.877766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d2798e-483d-4c7a-aaae-1a33f6d63584 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:09.878066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d2798e-483d-4c7a-aaae-1a33f6d63584 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:09.880290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d2798e-483d-4c7a-aaae-1a33f6d63584 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:09.911109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c0d2798e-483d-4c7a-aaae-1a33f6d63584 None None] MFA Rules not processed for user `b9a543a8b4f94fa7bdcc0a3b7c52396e`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:09.913170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c0d2798e-483d-4c7a-aaae-1a33f6d63584 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:10.016775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 952/1890] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:09 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:10.018431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3d6549-91bc-45c7-ac9a-31b88ac402ac admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:10.168747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3d6549-91bc-45c7-ac9a-31b88ac402ac admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:10.179089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9e3b53-b891-46fe-9476-56896f7dd2a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:10.182501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9e3b53-b891-46fe-9476-56896f7dd2a7 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:10.184888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9e3b53-b891-46fe-9476-56896f7dd2a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:10.185557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b9e3b53-b891-46fe-9476-56896f7dd2a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:10.186859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9e3b53-b891-46fe-9476-56896f7dd2a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:10.189798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b9e3b53-b891-46fe-9476-56896f7dd2a7 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:10.226465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 939/1891] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:10 2022] POST /identity/v3/projects => generated 379 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:10.232858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d2798e-483d-4c7a-aaae-1a33f6d63584 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:10.244583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d2798e-483d-4c7a-aaae-1a33f6d63584 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:10.319950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c956181-fb10-47a8-ac7f-737e0c014e01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:10.320729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c956181-fb10-47a8-ac7f-737e0c014e01 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:10.321090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c956181-fb10-47a8-ac7f-737e0c014e01 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:10.321380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c956181-fb10-47a8-ac7f-737e0c014e01 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:10.322062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c956181-fb10-47a8-ac7f-737e0c014e01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:10.323986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c956181-fb10-47a8-ac7f-737e0c014e01 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:10.355483 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 953/1892] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:10 2022] POST /identity/v3/users => generated 296 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:10.356754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9e3b53-b891-46fe-9476-56896f7dd2a7 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:10.368885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b9e3b53-b891-46fe-9476-56896f7dd2a7 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:10.373672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ad0a4f-6bd0-4e5b-b2c5-a1bea6c69d2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:10.374559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ad0a4f-6bd0-4e5b-b2c5-a1bea6c69d2f admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:10.374850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ad0a4f-6bd0-4e5b-b2c5-a1bea6c69d2f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:10.375132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ad0a4f-6bd0-4e5b-b2c5-a1bea6c69d2f admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:10.375835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ad0a4f-6bd0-4e5b-b2c5-a1bea6c69d2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:10.377551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ad0a4f-6bd0-4e5b-b2c5-a1bea6c69d2f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:10.388925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 940/1893] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:10 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:10.393296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c956181-fb10-47a8-ac7f-737e0c014e01 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:10.403322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c956181-fb10-47a8-ac7f-737e0c014e01 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:10.408062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e062104a-bf56-4473-b423-3a075b7268d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:10.408804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e062104a-bf56-4473-b423-3a075b7268d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:10.409117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e062104a-bf56-4473-b423-3a075b7268d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:10.409411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e062104a-bf56-4473-b423-3a075b7268d6 admin admin] PATH_INFO: `/v3/projects/ae3d4e55a03f41be96726252edca3411/users/60b6fd25a2d84b39a9ba223868f7a90f/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:10.426765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e062104a-bf56-4473-b423-3a075b7268d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae3d4e55a03f41be96726252edca3411, user_id=60b6fd25a2d84b39a9ba223868f7a90f, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:10.429466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e062104a-bf56-4473-b423-3a075b7268d6 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:10.443861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 954/1894] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:10 2022] PUT /identity/v3/projects/ae3d4e55a03f41be96726252edca3411/users/60b6fd25a2d84b39a9ba223868f7a90f/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:10.447944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ad0a4f-6bd0-4e5b-b2c5-a1bea6c69d2f admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:10.460644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ad0a4f-6bd0-4e5b-b2c5-a1bea6c69d2f admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:10.478918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:10.521682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc46e3c0-b00d-45a4-be79-1453b8615a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:10.522648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc46e3c0-b00d-45a4-be79-1453b8615a05 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:10.522939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc46e3c0-b00d-45a4-be79-1453b8615a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:10.523223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc46e3c0-b00d-45a4-be79-1453b8615a05 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:10.523933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc46e3c0-b00d-45a4-be79-1453b8615a05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:10.525589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc46e3c0-b00d-45a4-be79-1453b8615a05 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:10.536051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 941/1895] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:10 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:10.540139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e062104a-bf56-4473-b423-3a075b7268d6 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:10.550686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e062104a-bf56-4473-b423-3a075b7268d6 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:10.555304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511a1492-017d-4e3c-8562-964dcdabf19d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:10.556016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a1492-017d-4e3c-8562-964dcdabf19d admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:10.556294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a1492-017d-4e3c-8562-964dcdabf19d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:10.556583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a1492-017d-4e3c-8562-964dcdabf19d admin admin] PATH_INFO: `/v3/projects/ae3d4e55a03f41be96726252edca3411/users/60b6fd25a2d84b39a9ba223868f7a90f/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:10.558574 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511a1492-017d-4e3c-8562-964dcdabf19d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae3d4e55a03f41be96726252edca3411, user_id=60b6fd25a2d84b39a9ba223868f7a90f, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:10.561264 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511a1492-017d-4e3c-8562-964dcdabf19d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:10.573189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 955/1896] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:10 2022] PUT /identity/v3/projects/ae3d4e55a03f41be96726252edca3411/users/60b6fd25a2d84b39a9ba223868f7a90f/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:10.586497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685974bf-b81f-4eda-a120-02d0b07096b3 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:10.586891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685974bf-b81f-4eda-a120-02d0b07096b3 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:10.587238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685974bf-b81f-4eda-a120-02d0b07096b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:10.616670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-685974bf-b81f-4eda-a120-02d0b07096b3 None None] MFA Rules not processed for user `60b6fd25a2d84b39a9ba223868f7a90f`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:10.617894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-685974bf-b81f-4eda-a120-02d0b07096b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:10.696226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 942/1897] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:10 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:10.709487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511a1492-017d-4e3c-8562-964dcdabf19d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:10.711284 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-511a1492-017d-4e3c-8562-964dcdabf19d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:10.727889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:10.782408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511a1492-017d-4e3c-8562-964dcdabf19d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:10.786891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05a6c87-307d-4d2d-b87c-66b1f01bdf7a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:10.787527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05a6c87-307d-4d2d-b87c-66b1f01bdf7a service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:10.787818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05a6c87-307d-4d2d-b87c-66b1f01bdf7a service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:10.788104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05a6c87-307d-4d2d-b87c-66b1f01bdf7a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:10.800547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05a6c87-307d-4d2d-b87c-66b1f01bdf7a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:10.804210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05a6c87-307d-4d2d-b87c-66b1f01bdf7a service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:10.817778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 956/1898] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:10 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:11.315212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685974bf-b81f-4eda-a120-02d0b07096b3 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:11.335549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685974bf-b81f-4eda-a120-02d0b07096b3 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:11.343284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef63d323-c28e-4f37-9ce2-6af31abaa1e8 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60b6fd25a2d84b39a9ba223868f7a90f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3d4e55a03f41be96726252edca3411', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:11.344940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef63d323-c28e-4f37-9ce2-6af31abaa1e8 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:11.346081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef63d323-c28e-4f37-9ce2-6af31abaa1e8 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:11.346607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef63d323-c28e-4f37-9ce2-6af31abaa1e8 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:11.348224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef63d323-c28e-4f37-9ce2-6af31abaa1e8 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:11.348770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef63d323-c28e-4f37-9ce2-6af31abaa1e8 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:11.351206 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef63d323-c28e-4f37-9ce2-6af31abaa1e8 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:11.367266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 943/1899] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:48:11.371503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05a6c87-307d-4d2d-b87c-66b1f01bdf7a service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:11.383769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05a6c87-307d-4d2d-b87c-66b1f01bdf7a service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:11.390039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b4b5e8-868e-488c-9f10-bd3ce1a0d01d tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60b6fd25a2d84b39a9ba223868f7a90f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3d4e55a03f41be96726252edca3411', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:11.390729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b4b5e8-868e-488c-9f10-bd3ce1a0d01d tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:11.391024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b4b5e8-868e-488c-9f10-bd3ce1a0d01d tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:11.391308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b4b5e8-868e-488c-9f10-bd3ce1a0d01d tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:11.391986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b4b5e8-868e-488c-9f10-bd3ce1a0d01d tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:11.394479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b4b5e8-868e-488c-9f10-bd3ce1a0d01d tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:11.430111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 957/1900] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:11 2022] POST /identity/v3/projects => generated 354 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:11.434281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef63d323-c28e-4f37-9ce2-6af31abaa1e8 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:11.447277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef63d323-c28e-4f37-9ce2-6af31abaa1e8 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:11.522641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad929df0-6f23-48f2-979b-389d6e4ce003 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60b6fd25a2d84b39a9ba223868f7a90f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3d4e55a03f41be96726252edca3411', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:11.523292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad929df0-6f23-48f2-979b-389d6e4ce003 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:11.523652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad929df0-6f23-48f2-979b-389d6e4ce003 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:11.524007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad929df0-6f23-48f2-979b-389d6e4ce003 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:11.524870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad929df0-6f23-48f2-979b-389d6e4ce003 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:11.525201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad929df0-6f23-48f2-979b-389d6e4ce003 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:11.526858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad929df0-6f23-48f2-979b-389d6e4ce003 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:11.541113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 944/1901] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:48:11.591961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b4b5e8-868e-488c-9f10-bd3ce1a0d01d tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:11.603322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b4b5e8-868e-488c-9f10-bd3ce1a0d01d tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:11.608164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60b6fd25a2d84b39a9ba223868f7a90f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae3d4e55a03f41be96726252edca3411', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:11.608944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:11.609239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:11.609536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] PATH_INFO: `/v3/projects/655fe652427640f99c3a16b97b20ad5d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:11.610712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: Authorizing `identity:delete_project(project_id=655fe652427640f99c3a16b97b20ad5d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:11.612657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:11.665407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '655fe652427640f99c3a16b97b20ad5d', 'request_id': 'req-31da510f-4b3d-404e-897b-62717b2d70b7'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:11.673282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '655fe652427640f99c3a16b97b20ad5d', 'request_id': 'req-31da510f-4b3d-404e-897b-62717b2d70b7'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:11.680434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '655fe652427640f99c3a16b97b20ad5d', 'request_id': 'req-31da510f-4b3d-404e-897b-62717b2d70b7'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:11.690562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] The token cache is being invalidate because project 655fe652427640f99c3a16b97b20ad5d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:11.691376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:11.692222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-31da510f-4b3d-404e-897b-62717b2d70b7 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:11.693820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 958/1902] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:11 2022] DELETE /identity/v3/projects/655fe652427640f99c3a16b97b20ad5d => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:12.107528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad929df0-6f23-48f2-979b-389d6e4ce003 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:12.109489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ad929df0-6f23-48f2-979b-389d6e4ce003 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:12.165930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:12.263509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad929df0-6f23-48f2-979b-389d6e4ce003 tempest-VolumeQuotaClassesTest-666272593 tempest-VolumeQuotaClassesTest-666272593-project-admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:12.387192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:12.388149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:12.388438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:12.388736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] PATH_INFO: `/v3/users/b9a543a8b4f94fa7bdcc0a3b7c52396e` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:12.390972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9a543a8b4f94fa7bdcc0a3b7c52396e)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:12.408146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:12.528942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9a543a8b4f94fa7bdcc0a3b7c52396e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:12.542196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d38d6d-1024-4173-b90b-f37ab67987e7 None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:12.542646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d38d6d-1024-4173-b90b-f37ab67987e7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:12.542765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d38d6d-1024-4173-b90b-f37ab67987e7 None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:12.544670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 959/1903] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:48:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 29 04:48:12.547460 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9a543a8b4f94fa7bdcc0a3b7c52396e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:12.548305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9a543a8b4f94fa7bdcc0a3b7c52396e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:12.549105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9a543a8b4f94fa7bdcc0a3b7c52396e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:12.572365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dfd59c58-af9d-4818-943c-6ba291c02ae5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9a543a8b4f94fa7bdcc0a3b7c52396e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:12.580370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb86d2c2-eb50-4624-846c-833b1c154b3c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:12.580913 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb86d2c2-eb50-4624-846c-833b1c154b3c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:12.581431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb86d2c2-eb50-4624-846c-833b1c154b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:12.640023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-bb86d2c2-eb50-4624-846c-833b1c154b3c None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:12.646864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-bb86d2c2-eb50-4624-846c-833b1c154b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:12.651011 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 945/1904] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:12 2022] DELETE /identity/v3/users/b9a543a8b4f94fa7bdcc0a3b7c52396e => generated 0 bytes in 541 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:12.652764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69ef964-6593-4b32-a783-249bce7627a0 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:12.654296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69ef964-6593-4b32-a783-249bce7627a0 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:12.658312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69ef964-6593-4b32-a783-249bce7627a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:12.720516 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-a69ef964-6593-4b32-a783-249bce7627a0 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:12.722582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a69ef964-6593-4b32-a783-249bce7627a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:12.830425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:12.865788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 946/1905] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:12.866013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 960/1906] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:48:12 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:12.902349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69ef964-6593-4b32-a783-249bce7627a0 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:12.904973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a69ef964-6593-4b32-a783-249bce7627a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:12.922127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb86d2c2-eb50-4624-846c-833b1c154b3c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:12.926547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-bb86d2c2-eb50-4624-846c-833b1c154b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:12.955754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb86d2c2-eb50-4624-846c-833b1c154b3c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:12.966650 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0c64b5-4c01-49f5-818b-07820a0ae83f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:12.968568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0c64b5-4c01-49f5-818b-07820a0ae83f service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:12.969119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0c64b5-4c01-49f5-818b-07820a0ae83f service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:12.969587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0c64b5-4c01-49f5-818b-07820a0ae83f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:12.978633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:12.992704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0c64b5-4c01-49f5-818b-07820a0ae83f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:12.994346 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0c64b5-4c01-49f5-818b-07820a0ae83f service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:13.008776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69ef964-6593-4b32-a783-249bce7627a0 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:13.009324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 961/1907] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:12 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:13.013751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87454cd-cd26-4475-8f2a-884f3b50d3ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:13.014626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87454cd-cd26-4475-8f2a-884f3b50d3ac service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:13.014918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87454cd-cd26-4475-8f2a-884f3b50d3ac service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:13.015210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87454cd-cd26-4475-8f2a-884f3b50d3ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:13.026150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87454cd-cd26-4475-8f2a-884f3b50d3ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:13.027633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a87454cd-cd26-4475-8f2a-884f3b50d3ac service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:13.052700 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 947/1908] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:13.361920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0c64b5-4c01-49f5-818b-07820a0ae83f service nova] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:13.364026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3b0c64b5-4c01-49f5-818b-07820a0ae83f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:13.379492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0c64b5-4c01-49f5-818b-07820a0ae83f service nova] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:13.384689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:13.385444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:13.385733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:13.386057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] PATH_INFO: `/v3/projects/6dde749e999748eca35eda09181f0766` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:13.387290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6dde749e999748eca35eda09181f0766)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:13.389387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:13.440564 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6dde749e999748eca35eda09181f0766', 'request_id': 'req-c55b97a7-2f8f-4ced-88d4-df7a8500db11'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.448305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6dde749e999748eca35eda09181f0766', 'request_id': 'req-c55b97a7-2f8f-4ced-88d4-df7a8500db11'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.457284 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6dde749e999748eca35eda09181f0766', 'request_id': 'req-c55b97a7-2f8f-4ced-88d4-df7a8500db11'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.469799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] The token cache is being invalidate because project 6dde749e999748eca35eda09181f0766 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:13.470094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.470751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.472678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 962/1909] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:13 2022] DELETE /identity/v3/projects/6dde749e999748eca35eda09181f0766 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:13.476414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87454cd-cd26-4475-8f2a-884f3b50d3ac service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:13.486304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87454cd-cd26-4475-8f2a-884f3b50d3ac service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:13.546740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:13.547650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:13.548093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:13.548540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] PATH_INFO: `/v3/users/60b6fd25a2d84b39a9ba223868f7a90f` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:13.549998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] RBAC: Authorizing `identity:delete_user(user_id=60b6fd25a2d84b39a9ba223868f7a90f)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:13.552648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:13.608029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '60b6fd25a2d84b39a9ba223868f7a90f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.618334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '60b6fd25a2d84b39a9ba223868f7a90f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.619175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '60b6fd25a2d84b39a9ba223868f7a90f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.620175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '60b6fd25a2d84b39a9ba223868f7a90f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.629100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '60b6fd25a2d84b39a9ba223868f7a90f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.651299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 948/1910] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:13 2022] DELETE /identity/v3/users/60b6fd25a2d84b39a9ba223868f7a90f => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:13.754623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:13.756097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:13.773374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c55b97a7-2f8f-4ced-88d4-df7a8500db11 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:13.841138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:13.842590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:13.865634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:13.867641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:13.868096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:13.868640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] PATH_INFO: `/v3/users/979b92659a6d4615a8df21688ef6e420` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:13.870437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] RBAC: Authorizing `identity:delete_user(user_id=979b92659a6d4615a8df21688ef6e420)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:13.873786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:13.882383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:13.960782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '979b92659a6d4615a8df21688ef6e420'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.961489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '979b92659a6d4615a8df21688ef6e420'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.962123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '979b92659a6d4615a8df21688ef6e420'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:13.992191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc0fcfb-97ba-44ac-98a0-085d9c47315d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:13.996226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '979b92659a6d4615a8df21688ef6e420'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.006268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:14.007140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.007697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.008125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] PATH_INFO: `/v3/users/e488ec5312a94f0d9b2477bada393303` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.009721 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e488ec5312a94f0d9b2477bada393303)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:14.010619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e6ac7e3a-912c-44f3-b976-ec3b25ad4414 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '979b92659a6d4615a8df21688ef6e420'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.012071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:14.017982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:14.033106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 963/1911] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:13 2022] DELETE /identity/v3/users/979b92659a6d4615a8df21688ef6e420 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:14.039649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cfd53a-71b4-4d3a-a99e-47aee69ee204 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.040490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cfd53a-71b4-4d3a-a99e-47aee69ee204 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.040698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cfd53a-71b4-4d3a-a99e-47aee69ee204 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.068679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-12cfd53a-71b4-4d3a-a99e-47aee69ee204 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:14.070271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-12cfd53a-71b4-4d3a-a99e-47aee69ee204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:14.138513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e488ec5312a94f0d9b2477bada393303'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.165209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e488ec5312a94f0d9b2477bada393303'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.166045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e488ec5312a94f0d9b2477bada393303'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.167167 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e488ec5312a94f0d9b2477bada393303'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.176229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 964/1912] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:14.177318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cfd53a-71b4-4d3a-a99e-47aee69ee204 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:14.179035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-12cfd53a-71b4-4d3a-a99e-47aee69ee204 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:14.187907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e488ec5312a94f0d9b2477bada393303'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.218570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cfd53a-71b4-4d3a-a99e-47aee69ee204 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:14.246351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 949/1913] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:13 2022] DELETE /identity/v3/users/e488ec5312a94f0d9b2477bada393303 => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:14.247048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:14.248908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:14.249750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.249872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:14.250043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.250331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] PATH_INFO: `/v3/projects/ae3d4e55a03f41be96726252edca3411` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.253472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae3d4e55a03f41be96726252edca3411)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:14.256800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:14.367313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae3d4e55a03f41be96726252edca3411', 'request_id': 'req-711b85b5-7e53-4939-9cce-682f620370e6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.387454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae3d4e55a03f41be96726252edca3411', 'request_id': 'req-711b85b5-7e53-4939-9cce-682f620370e6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.404545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae3d4e55a03f41be96726252edca3411', 'request_id': 'req-711b85b5-7e53-4939-9cce-682f620370e6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.404902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beff693e-d055-45e6-8e60-ba868266bf4e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:14.421927 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] The token cache is being invalidate because project ae3d4e55a03f41be96726252edca3411 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:14.422501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.423724 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-711b85b5-7e53-4939-9cce-682f620370e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:14.425899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 965/1914] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:14 2022] DELETE /identity/v3/projects/ae3d4e55a03f41be96726252edca3411 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:14.427536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9c668-6137-432d-8759-d5326a25fc87 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.428353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9c668-6137-432d-8759-d5326a25fc87 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.428875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9c668-6137-432d-8759-d5326a25fc87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.474081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-92b9c668-6137-432d-8759-d5326a25fc87 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:14.477818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-92b9c668-6137-432d-8759-d5326a25fc87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:14.478402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:14.489984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb4e668-15bb-4ee2-874a-a68c8122b930 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:14.491655 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb4e668-15bb-4ee2-874a-a68c8122b930 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.492238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb4e668-15bb-4ee2-874a-a68c8122b930 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.493574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb4e668-15bb-4ee2-874a-a68c8122b930 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.497797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3eb4e668-15bb-4ee2-874a-a68c8122b930 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:14.573852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 966/1915] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:14.575387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d48ef48-ed35-44ea-a4b7-1ea4c62f5edd None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.575683 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d48ef48-ed35-44ea-a4b7-1ea4c62f5edd None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.575965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d48ef48-ed35-44ea-a4b7-1ea4c62f5edd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.587040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb4e668-15bb-4ee2-874a-a68c8122b930 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:14.588858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb4e668-15bb-4ee2-874a-a68c8122b930 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:14.601660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 950/1916] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:14.602898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2d48ef48-ed35-44ea-a4b7-1ea4c62f5edd None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:14.604248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2d48ef48-ed35-44ea-a4b7-1ea4c62f5edd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:14.605663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb4e668-15bb-4ee2-874a-a68c8122b930 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:14.608415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3eb4e668-15bb-4ee2-874a-a68c8122b930 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:14.612834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 967/1917] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:14.621600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd115ca1-b3bf-41f9-b986-10147232a6c6 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.621915 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd115ca1-b3bf-41f9-b986-10147232a6c6 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.622200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd115ca1-b3bf-41f9-b986-10147232a6c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.622753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb4e668-15bb-4ee2-874a-a68c8122b930 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:14.627885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6d4e55-573b-4fde-b3e7-99f4fc56178d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:14.631535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6d4e55-573b-4fde-b3e7-99f4fc56178d service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.633182 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6d4e55-573b-4fde-b3e7-99f4fc56178d service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.633679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6d4e55-573b-4fde-b3e7-99f4fc56178d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.648163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6d4e55-573b-4fde-b3e7-99f4fc56178d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:14.650305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-bd115ca1-b3bf-41f9-b986-10147232a6c6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:14.650458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6d4e55-573b-4fde-b3e7-99f4fc56178d service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:14.652149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-bd115ca1-b3bf-41f9-b986-10147232a6c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:14.662501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 968/1918] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:14.666898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd115ca1-b3bf-41f9-b986-10147232a6c6 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:14.675202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 951/1919] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:14.693981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd115ca1-b3bf-41f9-b986-10147232a6c6 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:14.709481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6314b803-b91f-4956-aa7e-e98a2ae9e0c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:14.711685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6314b803-b91f-4956-aa7e-e98a2ae9e0c4 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.712007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6314b803-b91f-4956-aa7e-e98a2ae9e0c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.714197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6314b803-b91f-4956-aa7e-e98a2ae9e0c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.715043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6314b803-b91f-4956-aa7e-e98a2ae9e0c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:14.715366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6314b803-b91f-4956-aa7e-e98a2ae9e0c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:14.721307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6314b803-b91f-4956-aa7e-e98a2ae9e0c4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:14.724390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:14.744778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 969/1920] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:48:14.751133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6d4e55-573b-4fde-b3e7-99f4fc56178d service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:14.772057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6d4e55-573b-4fde-b3e7-99f4fc56178d service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:14.782866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f9bf15-47ad-4584-b3cf-fba036d23fce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:14.784131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f9bf15-47ad-4584-b3cf-fba036d23fce admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.784892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f9bf15-47ad-4584-b3cf-fba036d23fce admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.787374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f9bf15-47ad-4584-b3cf-fba036d23fce admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.788474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f9bf15-47ad-4584-b3cf-fba036d23fce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:14.790885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f9bf15-47ad-4584-b3cf-fba036d23fce admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:14.832852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 952/1921] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:14 2022] POST /identity/v3/projects => generated 393 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:14.839989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6314b803-b91f-4956-aa7e-e98a2ae9e0c4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:14.863684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6314b803-b91f-4956-aa7e-e98a2ae9e0c4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:14.940268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c243751-6da9-4689-a946-a8b3d4dd2402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:14.941519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c243751-6da9-4689-a946-a8b3d4dd2402 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.941812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c243751-6da9-4689-a946-a8b3d4dd2402 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.942136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c243751-6da9-4689-a946-a8b3d4dd2402 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.942945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c243751-6da9-4689-a946-a8b3d4dd2402 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:14.944964 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c243751-6da9-4689-a946-a8b3d4dd2402 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:14.971496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 970/1922] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:14 2022] POST /identity/v3/users => generated 297 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:14.975904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f9bf15-47ad-4584-b3cf-fba036d23fce admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:14.988153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f9bf15-47ad-4584-b3cf-fba036d23fce admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:14.994244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a881cb-0718-459d-9cb7-64f47b686fd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:14.995448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a881cb-0718-459d-9cb7-64f47b686fd4 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:14.995805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a881cb-0718-459d-9cb7-64f47b686fd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:14.996159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a881cb-0718-459d-9cb7-64f47b686fd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:14.997391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a881cb-0718-459d-9cb7-64f47b686fd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:14.999660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a881cb-0718-459d-9cb7-64f47b686fd4 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:15.016702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 953/1923] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:14 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:15.018122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c243751-6da9-4689-a946-a8b3d4dd2402 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:15.032292 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c243751-6da9-4689-a946-a8b3d4dd2402 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:15.038655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea230f33-0855-4975-89f5-66ca7a6f7378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:15.039650 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea230f33-0855-4975-89f5-66ca7a6f7378 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:15.039945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea230f33-0855-4975-89f5-66ca7a6f7378 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:15.040227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea230f33-0855-4975-89f5-66ca7a6f7378 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:15.041053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea230f33-0855-4975-89f5-66ca7a6f7378 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:15.042680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea230f33-0855-4975-89f5-66ca7a6f7378 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:15.061527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 971/1924] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:15 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:15.062831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a881cb-0718-459d-9cb7-64f47b686fd4 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:15.096552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a881cb-0718-459d-9cb7-64f47b686fd4 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:15.105583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b0d509-a2b2-4cff-a19c-1057b8a4c06e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:15.108101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b0d509-a2b2-4cff-a19c-1057b8a4c06e admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:15.108742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b0d509-a2b2-4cff-a19c-1057b8a4c06e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:15.109258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b0d509-a2b2-4cff-a19c-1057b8a4c06e admin admin] PATH_INFO: `/v3/projects/ab62f02c8cb14148aaca480aaf75193e/users/6d61f07b0cab409cbb3d2c93b6f987d7/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:15.144421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b0d509-a2b2-4cff-a19c-1057b8a4c06e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab62f02c8cb14148aaca480aaf75193e, user_id=6d61f07b0cab409cbb3d2c93b6f987d7, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:15.149639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b0d509-a2b2-4cff-a19c-1057b8a4c06e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:15.171776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 954/1925] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:15 2022] PUT /identity/v3/projects/ab62f02c8cb14148aaca480aaf75193e/users/6d61f07b0cab409cbb3d2c93b6f987d7/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:15.184466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d4d083-c28d-4c02-a6bb-15e1419cf01d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:15.184921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d4d083-c28d-4c02-a6bb-15e1419cf01d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:15.185300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d4d083-c28d-4c02-a6bb-15e1419cf01d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:15.218402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-25d4d083-c28d-4c02-a6bb-15e1419cf01d None None] MFA Rules not processed for user `6d61f07b0cab409cbb3d2c93b6f987d7`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:15.219789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-25d4d083-c28d-4c02-a6bb-15e1419cf01d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:15.316309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 972/1926] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:15 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:15.321286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b0d509-a2b2-4cff-a19c-1057b8a4c06e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:15.418673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b0d509-a2b2-4cff-a19c-1057b8a4c06e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:15.426010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a4d748-1537-43a2-893b-b68b545776fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:15.427840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a4d748-1537-43a2-893b-b68b545776fd admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:15.427840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a4d748-1537-43a2-893b-b68b545776fd admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:15.427840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a4d748-1537-43a2-893b-b68b545776fd admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:15.429099 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a4d748-1537-43a2-893b-b68b545776fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:15.431685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90a4d748-1537-43a2-893b-b68b545776fd admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:15.493154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d4d083-c28d-4c02-a6bb-15e1419cf01d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:15.499456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 955/1927] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:15 2022] POST /identity/v3/projects => generated 389 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:15.508994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d4d083-c28d-4c02-a6bb-15e1419cf01d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:15.676450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d662f6-7bb7-486e-9136-985144d9c0b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:15.677085 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d662f6-7bb7-486e-9136-985144d9c0b2 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:15.677378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d662f6-7bb7-486e-9136-985144d9c0b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:15.677661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d662f6-7bb7-486e-9136-985144d9c0b2 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:15.678353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d662f6-7bb7-486e-9136-985144d9c0b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:15.680488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d662f6-7bb7-486e-9136-985144d9c0b2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:15.707887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:15.756805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 973/1928] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:15 2022] POST /identity/v3/users => generated 301 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:15.759932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a4d748-1537-43a2-893b-b68b545776fd admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:15.772832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a4d748-1537-43a2-893b-b68b545776fd admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:15.782083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5de2be-8c9f-407c-beef-e78d262dee42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:15.782727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5de2be-8c9f-407c-beef-e78d262dee42 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:15.783135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5de2be-8c9f-407c-beef-e78d262dee42 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:15.783699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5de2be-8c9f-407c-beef-e78d262dee42 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:15.784297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5de2be-8c9f-407c-beef-e78d262dee42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:15.787071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5de2be-8c9f-407c-beef-e78d262dee42 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:15.793407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:15.805513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 956/1929] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:15 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:15.807516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d662f6-7bb7-486e-9136-985144d9c0b2 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:15.822525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d662f6-7bb7-486e-9136-985144d9c0b2 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:15.829697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68314a2-a4b1-457c-9609-75cb1eafb492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:15.830267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68314a2-a4b1-457c-9609-75cb1eafb492 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:15.830719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68314a2-a4b1-457c-9609-75cb1eafb492 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:15.831127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68314a2-a4b1-457c-9609-75cb1eafb492 admin admin] PATH_INFO: `/v3/projects/54940752312842149aae35037e1f7651/users/6bb39df7be6b4456a87a99b3bc36fa4d/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:15.864946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68314a2-a4b1-457c-9609-75cb1eafb492 admin admin] RBAC: Authorizing `identity:create_grant(project_id=54940752312842149aae35037e1f7651, user_id=6bb39df7be6b4456a87a99b3bc36fa4d, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:15.875248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68314a2-a4b1-457c-9609-75cb1eafb492 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:15.904519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 974/1930] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:15 2022] PUT /identity/v3/projects/54940752312842149aae35037e1f7651/users/6bb39df7be6b4456a87a99b3bc36fa4d/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:15.908553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5de2be-8c9f-407c-beef-e78d262dee42 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:15.952677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5de2be-8c9f-407c-beef-e78d262dee42 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:15.981852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68314a2-a4b1-457c-9609-75cb1eafb492 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:15.986080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c68314a2-a4b1-457c-9609-75cb1eafb492 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:16.071905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb80f50-e93e-44f3-b41d-a0f3e3364222 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:16.073972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb80f50-e93e-44f3-b41d-a0f3e3364222 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:16.074587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb80f50-e93e-44f3-b41d-a0f3e3364222 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:16.075146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb80f50-e93e-44f3-b41d-a0f3e3364222 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:16.076361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb80f50-e93e-44f3-b41d-a0f3e3364222 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:16.079873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb80f50-e93e-44f3-b41d-a0f3e3364222 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:16.090545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68314a2-a4b1-457c-9609-75cb1eafb492 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:16.095435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:16.096096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:16.096385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:16.096722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] PATH_INFO: `/v3/projects/0a708949485a4fa1844ceab95079774d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:16.098579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a708949485a4fa1844ceab95079774d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:16.100500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:16.109414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 957/1931] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:15 2022] GET /identity/v3/roles => generated 1420 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:16.110509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb80f50-e93e-44f3-b41d-a0f3e3364222 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:16.113999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-dbb80f50-e93e-44f3-b41d-a0f3e3364222 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:16.131316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb80f50-e93e-44f3-b41d-a0f3e3364222 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:16.138993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:16.140463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:16.141036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:16.141313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] PATH_INFO: `/v3/projects/c38ef094040b413cb1a7c37a4202c758` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:16.142734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c38ef094040b413cb1a7c37a4202c758)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:16.145229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:16.225895 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a708949485a4fa1844ceab95079774d', 'request_id': 'req-55f6fb4d-dbcd-4474-83af-3b53359d4d77'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.244791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a708949485a4fa1844ceab95079774d', 'request_id': 'req-55f6fb4d-dbcd-4474-83af-3b53359d4d77'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.258843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a708949485a4fa1844ceab95079774d', 'request_id': 'req-55f6fb4d-dbcd-4474-83af-3b53359d4d77'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.261939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c38ef094040b413cb1a7c37a4202c758', 'request_id': 'req-1e3580bf-2205-4a62-8674-8a11198c8d27'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.278171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c38ef094040b413cb1a7c37a4202c758', 'request_id': 'req-1e3580bf-2205-4a62-8674-8a11198c8d27'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.282579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] The token cache is being invalidate because project 0a708949485a4fa1844ceab95079774d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:16.283394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.286450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.292717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 975/1932] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:15 2022] DELETE /identity/v3/projects/0a708949485a4fa1844ceab95079774d => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:16.293566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c38ef094040b413cb1a7c37a4202c758', 'request_id': 'req-1e3580bf-2205-4a62-8674-8a11198c8d27'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.294714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:16.297398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:16.322839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] The token cache is being invalidate because project c38ef094040b413cb1a7c37a4202c758 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:16.323134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.323922 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.327581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 958/1933] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:16 2022] DELETE /identity/v3/projects/c38ef094040b413cb1a7c37a4202c758 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:16.330379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:16.333823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:16.416002 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:INSERT=4,keystone:SELECT=89,keystone:DELETE=7 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:16.422676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f6fb4d-dbcd-4474-83af-3b53359d4d77 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:16.431605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084badaa-5560-4808-8234-d91313386cef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:16.432885 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084badaa-5560-4808-8234-d91313386cef admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:16.433353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084badaa-5560-4808-8234-d91313386cef admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:16.433875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084badaa-5560-4808-8234-d91313386cef admin admin] PATH_INFO: `/v3/projects/54940752312842149aae35037e1f7651/users/6bb39df7be6b4456a87a99b3bc36fa4d/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:16.436636 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084badaa-5560-4808-8234-d91313386cef admin admin] RBAC: Authorizing `identity:create_grant(project_id=54940752312842149aae35037e1f7651, user_id=6bb39df7be6b4456a87a99b3bc36fa4d, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:16.437572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3580bf-2205-4a62-8674-8a11198c8d27 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:16.440356 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084badaa-5560-4808-8234-d91313386cef admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:16.444782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:16.445561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:16.445950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:16.446337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] PATH_INFO: `/v3/users/714991ba02164f5fb928e96c2d74d02c` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:16.447835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] RBAC: Authorizing `identity:delete_user(user_id=714991ba02164f5fb928e96c2d74d02c)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:16.450374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:16.454140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 976/1934] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:16 2022] PUT /identity/v3/projects/54940752312842149aae35037e1f7651/users/6bb39df7be6b4456a87a99b3bc36fa4d/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:16.455183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084badaa-5560-4808-8234-d91313386cef admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:16.470547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-084badaa-5560-4808-8234-d91313386cef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:16.480771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:16.537724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '714991ba02164f5fb928e96c2d74d02c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.549147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '714991ba02164f5fb928e96c2d74d02c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.551034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '714991ba02164f5fb928e96c2d74d02c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.552435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '714991ba02164f5fb928e96c2d74d02c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.560743 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-73c4cc74-e0da-4cc3-a3fe-a21f5abe9e04 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '714991ba02164f5fb928e96c2d74d02c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.571629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084badaa-5560-4808-8234-d91313386cef admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:16.576696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:16.577342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:16.577635 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:16.577914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] PATH_INFO: `/v3/users/04a8f3c55a5748768a65f65f82210198` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:16.579117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] RBAC: Authorizing `identity:delete_user(user_id=04a8f3c55a5748768a65f65f82210198)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:16.581078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:16.586102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 959/1935] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:16 2022] DELETE /identity/v3/users/714991ba02164f5fb928e96c2d74d02c => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:16.587653 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee893da-471b-4343-af9d-7efa64b6dfaf None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:16.588094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee893da-471b-4343-af9d-7efa64b6dfaf None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:16.588840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee893da-471b-4343-af9d-7efa64b6dfaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:16.627339 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0ee893da-471b-4343-af9d-7efa64b6dfaf None None] MFA Rules not processed for user `6bb39df7be6b4456a87a99b3bc36fa4d`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:16.629916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0ee893da-471b-4343-af9d-7efa64b6dfaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:16.645530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04a8f3c55a5748768a65f65f82210198'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.646275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04a8f3c55a5748768a65f65f82210198'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.647135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04a8f3c55a5748768a65f65f82210198'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.664297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04a8f3c55a5748768a65f65f82210198'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.672944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04a8f3c55a5748768a65f65f82210198'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:16.695811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 977/1936] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:16 2022] DELETE /identity/v3/users/04a8f3c55a5748768a65f65f82210198 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:16.775060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 960/1937] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:16 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:16.790594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:16.792629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:16.890705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0fbc6d-a597-4288-9711-f706cab2d41b admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:16.894540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5645fe85-5967-4547-b317-5beccad99d4f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:16.895260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645fe85-5967-4547-b317-5beccad99d4f service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:16.895645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645fe85-5967-4547-b317-5beccad99d4f service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:16.895992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5645fe85-5967-4547-b317-5beccad99d4f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:16.930194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:16.994569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5645fe85-5967-4547-b317-5beccad99d4f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:16.996026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5645fe85-5967-4547-b317-5beccad99d4f service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:17.009014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 978/1938] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:16 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:17.020510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee893da-471b-4343-af9d-7efa64b6dfaf None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:17.021987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0ee893da-471b-4343-af9d-7efa64b6dfaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:17.122167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5645fe85-5967-4547-b317-5beccad99d4f service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:17.123875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5645fe85-5967-4547-b317-5beccad99d4f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:17.142683 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5645fe85-5967-4547-b317-5beccad99d4f service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:17.143215 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee893da-471b-4343-af9d-7efa64b6dfaf None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:17.150334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2f3570-8367-4bd1-8442-0a9cd8f261b8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:17.151302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2f3570-8367-4bd1-8442-0a9cd8f261b8 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.151430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2f3570-8367-4bd1-8442-0a9cd8f261b8 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.151662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2f3570-8367-4bd1-8442-0a9cd8f261b8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.153803 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7f2f3570-8367-4bd1-8442-0a9cd8f261b8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:17.179054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:17.245801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:17.246459 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.246748 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.247030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] PATH_INFO: `/v3/projects/db96db215e1d4743baf316ebe875ddb4` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.249708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db96db215e1d4743baf316ebe875ddb4)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:17.251665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:17.273349 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2f3570-8367-4bd1-8442-0a9cd8f261b8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:17.274894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2f3570-8367-4bd1-8442-0a9cd8f261b8 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:17.289010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 979/1939] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:17 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:17.299677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2f3570-8367-4bd1-8442-0a9cd8f261b8 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:17.302842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7f2f3570-8367-4bd1-8442-0a9cd8f261b8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:17.323935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2f3570-8367-4bd1-8442-0a9cd8f261b8 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:17.332223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db96db215e1d4743baf316ebe875ddb4', 'request_id': 'req-ab3487be-076d-44f3-8fab-9a018e855f76'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:17.346834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db96db215e1d4743baf316ebe875ddb4', 'request_id': 'req-ab3487be-076d-44f3-8fab-9a018e855f76'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:17.357839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db96db215e1d4743baf316ebe875ddb4', 'request_id': 'req-ab3487be-076d-44f3-8fab-9a018e855f76'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:17.381566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] The token cache is being invalidate because project db96db215e1d4743baf316ebe875ddb4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:17.381860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:17.382776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ab3487be-076d-44f3-8fab-9a018e855f76 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:17.385785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 961/1940] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:17 2022] DELETE /identity/v3/projects/db96db215e1d4743baf316ebe875ddb4 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:17.396135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30cbb1f-7359-406a-9635-96f8648b3396 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.396435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30cbb1f-7359-406a-9635-96f8648b3396 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.396949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30cbb1f-7359-406a-9635-96f8648b3396 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.423743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:17.424780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.425989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e30cbb1f-7359-406a-9635-96f8648b3396 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:17.426540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.427069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] PATH_INFO: `/v3/projects/1995f18257274b8ab36d7e390a8f7e3c` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.427315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e30cbb1f-7359-406a-9635-96f8648b3396 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:17.429754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1995f18257274b8ab36d7e390a8f7e3c)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:17.431941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:17.438475 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 962/1941] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:17.451244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b809152-79e3-40e2-9299-1fb5a0fe44ad None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.451684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b809152-79e3-40e2-9299-1fb5a0fe44ad None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.451951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b809152-79e3-40e2-9299-1fb5a0fe44ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.493263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-2b809152-79e3-40e2-9299-1fb5a0fe44ad None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:17.497828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2b809152-79e3-40e2-9299-1fb5a0fe44ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:17.500940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1995f18257274b8ab36d7e390a8f7e3c', 'request_id': 'req-45f8a822-bc32-48ed-a75d-8295cea2d585'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:17.509060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1995f18257274b8ab36d7e390a8f7e3c', 'request_id': 'req-45f8a822-bc32-48ed-a75d-8295cea2d585'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:17.519742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1995f18257274b8ab36d7e390a8f7e3c', 'request_id': 'req-45f8a822-bc32-48ed-a75d-8295cea2d585'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:17.522601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:17.539581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] The token cache is being invalidate because project 1995f18257274b8ab36d7e390a8f7e3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:17.540082 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:17.541315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45f8a822-bc32-48ed-a75d-8295cea2d585 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:17.544037 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 980/1942] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:17 2022] DELETE /identity/v3/projects/1995f18257274b8ab36d7e390a8f7e3c => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:17.550286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ca33b8-d7ed-4d75-aba1-6492013cfd45 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.550737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ca33b8-d7ed-4d75-aba1-6492013cfd45 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.551018 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ca33b8-d7ed-4d75-aba1-6492013cfd45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.586981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-33ca33b8-d7ed-4d75-aba1-6492013cfd45 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:17.588980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-33ca33b8-d7ed-4d75-aba1-6492013cfd45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:17.657881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 963/1943] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:17.659487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b809152-79e3-40e2-9299-1fb5a0fe44ad None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:17.677701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b809152-79e3-40e2-9299-1fb5a0fe44ad None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:17.685572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c027d8-edea-4c6b-9501-4f9d73f49cb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:17.686586 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c027d8-edea-4c6b-9501-4f9d73f49cb2 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.687851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c027d8-edea-4c6b-9501-4f9d73f49cb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.688390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c027d8-edea-4c6b-9501-4f9d73f49cb2 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.689271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c027d8-edea-4c6b-9501-4f9d73f49cb2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:17.689821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c027d8-edea-4c6b-9501-4f9d73f49cb2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:17.692396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c027d8-edea-4c6b-9501-4f9d73f49cb2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:17.715641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 964/1944] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:48:17.719496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c027d8-edea-4c6b-9501-4f9d73f49cb2 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:17.723426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 981/1945] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:17.735439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25734b5-2975-45b1-a4de-8bf259a79052 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.735956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25734b5-2975-45b1-a4de-8bf259a79052 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.736348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25734b5-2975-45b1-a4de-8bf259a79052 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.744487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c027d8-edea-4c6b-9501-4f9d73f49cb2 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:17.763129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a25734b5-2975-45b1-a4de-8bf259a79052 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:17.765014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a25734b5-2975-45b1-a4de-8bf259a79052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:17.767816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e7fc52-fbc7-4605-95d9-894d5db37bf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:17.768314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e7fc52-fbc7-4605-95d9-894d5db37bf6 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.768652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e7fc52-fbc7-4605-95d9-894d5db37bf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.769626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e7fc52-fbc7-4605-95d9-894d5db37bf6 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.770422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e7fc52-fbc7-4605-95d9-894d5db37bf6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:17.772481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e7fc52-fbc7-4605-95d9-894d5db37bf6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:17.787931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 982/1946] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:17.794135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25734b5-2975-45b1-a4de-8bf259a79052 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:17.808077 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25734b5-2975-45b1-a4de-8bf259a79052 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:17.814720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c39cb9-acd8-45e2-920d-59c1a2ad87a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:17.816206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c39cb9-acd8-45e2-920d-59c1a2ad87a6 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.816919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c39cb9-acd8-45e2-920d-59c1a2ad87a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.817267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c39cb9-acd8-45e2-920d-59c1a2ad87a6 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.818062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c39cb9-acd8-45e2-920d-59c1a2ad87a6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:17.818443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c39cb9-acd8-45e2-920d-59c1a2ad87a6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:17.820186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c39cb9-acd8-45e2-920d-59c1a2ad87a6 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:17.837382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 983/1947] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:48:17.842814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c39cb9-acd8-45e2-920d-59c1a2ad87a6 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:17.850467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 965/1948] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:17 2022] POST /identity/v3/projects => generated 371 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:17.855445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e7fc52-fbc7-4605-95d9-894d5db37bf6 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:17.866820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c39cb9-acd8-45e2-920d-59c1a2ad87a6 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:17.895120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:17.934199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c34648-0ae0-4c8c-a0c2-c63c24d465ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:17.934824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c34648-0ae0-4c8c-a0c2-c63c24d465ee admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.935146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c34648-0ae0-4c8c-a0c2-c63c24d465ee admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.935430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c34648-0ae0-4c8c-a0c2-c63c24d465ee admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.936110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c34648-0ae0-4c8c-a0c2-c63c24d465ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:17.938113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c34648-0ae0-4c8c-a0c2-c63c24d465ee admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:17.950107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e7fc52-fbc7-4605-95d9-894d5db37bf6 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:17.955281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14983a96-2abf-46cd-b043-f0057d9a1a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:17.955901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14983a96-2abf-46cd-b043-f0057d9a1a88 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:17.956251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14983a96-2abf-46cd-b043-f0057d9a1a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:17.957271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14983a96-2abf-46cd-b043-f0057d9a1a88 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:17.958607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14983a96-2abf-46cd-b043-f0057d9a1a88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:17.962758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14983a96-2abf-46cd-b043-f0057d9a1a88 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:17.977161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 984/1949] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:17 2022] POST /identity/v3/projects => generated 377 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:17.978220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c34648-0ae0-4c8c-a0c2-c63c24d465ee admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:17.991706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 966/1950] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:17 2022] POST /identity/v3/users => generated 286 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:17.992698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14983a96-2abf-46cd-b043-f0057d9a1a88 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:18.007374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14983a96-2abf-46cd-b043-f0057d9a1a88 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:18.044916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:DELETE=2,keystone:SELECT=97,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:18.052755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c34648-0ae0-4c8c-a0c2-c63c24d465ee admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:18.061095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a05279-7a8d-4407-8f26-a581c9ca654d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:18.061731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a05279-7a8d-4407-8f26-a581c9ca654d admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.062021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a05279-7a8d-4407-8f26-a581c9ca654d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.062291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a05279-7a8d-4407-8f26-a581c9ca654d admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.063032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a05279-7a8d-4407-8f26-a581c9ca654d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:18.064935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a05279-7a8d-4407-8f26-a581c9ca654d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:18.076367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5eb6cfd-c8d2-422e-b52d-8b9019206f15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:18.078119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5eb6cfd-c8d2-422e-b52d-8b9019206f15 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.078448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5eb6cfd-c8d2-422e-b52d-8b9019206f15 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.078795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5eb6cfd-c8d2-422e-b52d-8b9019206f15 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.079745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5eb6cfd-c8d2-422e-b52d-8b9019206f15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:18.082125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5eb6cfd-c8d2-422e-b52d-8b9019206f15 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:18.091605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 985/1951] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:17 2022] POST /identity/v3/users => generated 289 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:18.092680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a05279-7a8d-4407-8f26-a581c9ca654d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:18.095144 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 967/1952] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:17 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:18.100133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5eb6cfd-c8d2-422e-b52d-8b9019206f15 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:18.103813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a05279-7a8d-4407-8f26-a581c9ca654d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:18.108673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d988af1-6b73-4447-8260-519c4c656d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:18.109297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d988af1-6b73-4447-8260-519c4c656d14 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.109791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d988af1-6b73-4447-8260-519c4c656d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.110318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d988af1-6b73-4447-8260-519c4c656d14 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.111251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d988af1-6b73-4447-8260-519c4c656d14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:18.111938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5eb6cfd-c8d2-422e-b52d-8b9019206f15 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:18.113686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d988af1-6b73-4447-8260-519c4c656d14 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:18.121455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b151303-7854-42ba-8da2-b49abefea99e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:18.122179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b151303-7854-42ba-8da2-b49abefea99e admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.122829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b151303-7854-42ba-8da2-b49abefea99e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.122829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b151303-7854-42ba-8da2-b49abefea99e admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.123733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b151303-7854-42ba-8da2-b49abefea99e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:18.125345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 986/1953] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:18.126520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b151303-7854-42ba-8da2-b49abefea99e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:18.129892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d988af1-6b73-4447-8260-519c4c656d14 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:18.141701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d988af1-6b73-4447-8260-519c4c656d14 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:18.141888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 968/1954] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:18 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:18.146384 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b151303-7854-42ba-8da2-b49abefea99e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:18.149269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719f1444-1dcd-44c0-9fa9-f43011be644f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:18.149997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719f1444-1dcd-44c0-9fa9-f43011be644f admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.150347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719f1444-1dcd-44c0-9fa9-f43011be644f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.150592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719f1444-1dcd-44c0-9fa9-f43011be644f admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.151294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719f1444-1dcd-44c0-9fa9-f43011be644f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:18.153007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719f1444-1dcd-44c0-9fa9-f43011be644f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:18.157261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b151303-7854-42ba-8da2-b49abefea99e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:18.162695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b7faf3-5b5c-4825-9e93-f87e042b5a30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:18.163361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7faf3-5b5c-4825-9e93-f87e042b5a30 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.163649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7faf3-5b5c-4825-9e93-f87e042b5a30 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.163934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7faf3-5b5c-4825-9e93-f87e042b5a30 admin admin] PATH_INFO: `/v3/projects/e52663d55be44e12b4aa939974e8e517/users/87f97b8b406e442b93cbdda090d7e9ff/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.164528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 987/1955] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:18 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:18.168238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719f1444-1dcd-44c0-9fa9-f43011be644f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:18.179003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719f1444-1dcd-44c0-9fa9-f43011be644f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:18.185397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b7faf3-5b5c-4825-9e93-f87e042b5a30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e52663d55be44e12b4aa939974e8e517, user_id=87f97b8b406e442b93cbdda090d7e9ff, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:18.185649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df8d401-4af1-4412-8955-44d779b263a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:18.186330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df8d401-4af1-4412-8955-44d779b263a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.186637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df8d401-4af1-4412-8955-44d779b263a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.187052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df8d401-4af1-4412-8955-44d779b263a6 admin admin] PATH_INFO: `/v3/projects/be01946c207e4739a5d95502633ce87d/users/5591e354be1e461fb5dcbe8a7ddb2cdd/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.189579 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b7faf3-5b5c-4825-9e93-f87e042b5a30 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:18.203953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df8d401-4af1-4412-8955-44d779b263a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=be01946c207e4739a5d95502633ce87d, user_id=5591e354be1e461fb5dcbe8a7ddb2cdd, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:18.206363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 969/1956] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:18 2022] PUT /identity/v3/projects/e52663d55be44e12b4aa939974e8e517/users/87f97b8b406e442b93cbdda090d7e9ff/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:18.207942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df8d401-4af1-4412-8955-44d779b263a6 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:18.211126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd706e-3dee-4b9a-a7b7-e56ab7ce3822 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.211364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd706e-3dee-4b9a-a7b7-e56ab7ce3822 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.211649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acd706e-3dee-4b9a-a7b7-e56ab7ce3822 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.225165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 988/1957] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:18 2022] PUT /identity/v3/projects/be01946c207e4739a5d95502633ce87d/users/5591e354be1e461fb5dcbe8a7ddb2cdd/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:18.231737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-8acd706e-3dee-4b9a-a7b7-e56ab7ce3822 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:18.233148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8acd706e-3dee-4b9a-a7b7-e56ab7ce3822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:18.238854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b74bc7d-b978-4083-844e-d9ecf1a129eb None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.239198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b74bc7d-b978-4083-844e-d9ecf1a129eb None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.239511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b74bc7d-b978-4083-844e-d9ecf1a129eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.273554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2b74bc7d-b978-4083-844e-d9ecf1a129eb None None] MFA Rules not processed for user `5591e354be1e461fb5dcbe8a7ddb2cdd`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:18.274955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2b74bc7d-b978-4083-844e-d9ecf1a129eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:18.333100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 970/1958] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:18.340024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acd706e-3dee-4b9a-a7b7-e56ab7ce3822 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:18.342205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8acd706e-3dee-4b9a-a7b7-e56ab7ce3822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:18.361199 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:18.368898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 989/1959] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:18 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:18.382846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b74bc7d-b978-4083-844e-d9ecf1a129eb None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:18.385062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2b74bc7d-b978-4083-844e-d9ecf1a129eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:18.430988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acd706e-3dee-4b9a-a7b7-e56ab7ce3822 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:18.435801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2a02d6-0a05-4b0c-9ea4-c4193474e25f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:18.436546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2a02d6-0a05-4b0c-9ea4-c4193474e25f service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.437086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2a02d6-0a05-4b0c-9ea4-c4193474e25f service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.437798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2a02d6-0a05-4b0c-9ea4-c4193474e25f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.457778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b74bc7d-b978-4083-844e-d9ecf1a129eb None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:18.462680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f142c07f-8851-418b-bfd2-4dce9fdcb274 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:18.463397 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f142c07f-8851-418b-bfd2-4dce9fdcb274 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:18.463759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f142c07f-8851-418b-bfd2-4dce9fdcb274 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:18.464113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f142c07f-8851-418b-bfd2-4dce9fdcb274 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:18.466753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2a02d6-0a05-4b0c-9ea4-c4193474e25f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:18.468374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2a02d6-0a05-4b0c-9ea4-c4193474e25f service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:18.474232 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f142c07f-8851-418b-bfd2-4dce9fdcb274 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:18.475857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f142c07f-8851-418b-bfd2-4dce9fdcb274 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:18.479669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 971/1960] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:18.484946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 990/1961] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:18 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:19.264811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c0b878-b1c8-4282-a8d3-dc632f10a5a7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:19.265312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c0b878-b1c8-4282-a8d3-dc632f10a5a7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:19.265404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c0b878-b1c8-4282-a8d3-dc632f10a5a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:19.297616 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-98c0b878-b1c8-4282-a8d3-dc632f10a5a7 None None] MFA Rules not processed for user `87f97b8b406e442b93cbdda090d7e9ff`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:19.298774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-98c0b878-b1c8-4282-a8d3-dc632f10a5a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:19.376590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 972/1962] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:19 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:19.388448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f142c07f-8851-418b-bfd2-4dce9fdcb274 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:19.398313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f142c07f-8851-418b-bfd2-4dce9fdcb274 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:19.402241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee738d2-d88e-43e3-8136-8015e006028d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:19.403039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee738d2-d88e-43e3-8136-8015e006028d service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:19.403481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee738d2-d88e-43e3-8136-8015e006028d service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:19.403887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee738d2-d88e-43e3-8136-8015e006028d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:19.415719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee738d2-d88e-43e3-8136-8015e006028d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:19.417360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee738d2-d88e-43e3-8136-8015e006028d service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:19.428942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 991/1963] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:19 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:22.916852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63e005-df03-4147-acba-0b47df193c3f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:22.917453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63e005-df03-4147-acba-0b47df193c3f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:22.917453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd63e005-df03-4147-acba-0b47df193c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:22.939556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-dd63e005-df03-4147-acba-0b47df193c3f None None] MFA Rules not processed for user `87f97b8b406e442b93cbdda090d7e9ff`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:22.941813 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-dd63e005-df03-4147-acba-0b47df193c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:22.949239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 973/1964] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:22 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:28.511066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee738d2-d88e-43e3-8136-8015e006028d service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:28.513148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-eee738d2-d88e-43e3-8136-8015e006028d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:28.528541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee738d2-d88e-43e3-8136-8015e006028d service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:28.540193 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:28.541495 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:28.542415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:28.542818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] PATH_INFO: `/v3/users/a76550322ec7408fa63ef23d64267daa` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:28.544765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a76550322ec7408fa63ef23d64267daa)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:28.547388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:28.576824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:28.592160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd63e005-df03-4147-acba-0b47df193c3f None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:28.593611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-dd63e005-df03-4147-acba-0b47df193c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:28.609081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd63e005-df03-4147-acba-0b47df193c3f None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:28.615399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:28.616760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:28.617675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:28.618527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] PATH_INFO: `/v3/users/6d61f07b0cab409cbb3d2c93b6f987d7` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:28.620470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d61f07b0cab409cbb3d2c93b6f987d7)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:28.623060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:28.644184 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a76550322ec7408fa63ef23d64267daa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:28.645454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a76550322ec7408fa63ef23d64267daa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:28.646087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a76550322ec7408fa63ef23d64267daa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:28.673728 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a76550322ec7408fa63ef23d64267daa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:28.681738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2246d7fc-c576-48b3-a152-88b5a6342777 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a76550322ec7408fa63ef23d64267daa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:28.688966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d61f07b0cab409cbb3d2c93b6f987d7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:28.696342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 992/1965] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:28 2022] DELETE /identity/v3/users/a76550322ec7408fa63ef23d64267daa => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:28.696708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d61f07b0cab409cbb3d2c93b6f987d7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:28.697614 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d61f07b0cab409cbb3d2c93b6f987d7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:28.698248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d61f07b0cab409cbb3d2c93b6f987d7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:28.700779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b478f998-5f84-4fbe-88ca-2669ae15d3b8 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:28.701076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b478f998-5f84-4fbe-88ca-2669ae15d3b8 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:28.701392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b478f998-5f84-4fbe-88ca-2669ae15d3b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:28.705405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8b2f4fdb-1d2d-4fbe-b640-be5441975367 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d61f07b0cab409cbb3d2c93b6f987d7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:28.723561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 974/1966] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:28 2022] DELETE /identity/v3/users/6d61f07b0cab409cbb3d2c93b6f987d7 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:28.723738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-b478f998-5f84-4fbe-88ca-2669ae15d3b8 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:28.725041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b478f998-5f84-4fbe-88ca-2669ae15d3b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:28.727748 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed2a7a-c455-4472-bef1-c5176601840b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:28.728038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed2a7a-c455-4472-bef1-c5176601840b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:28.728318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed2a7a-c455-4472-bef1-c5176601840b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:28.747044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-71ed2a7a-c455-4472-bef1-c5176601840b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:28.748238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-71ed2a7a-c455-4472-bef1-c5176601840b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:28.752130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:28.797388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 993/1967] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:28.808343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b478f998-5f84-4fbe-88ca-2669ae15d3b8 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:28.809808 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b478f998-5f84-4fbe-88ca-2669ae15d3b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:28.824714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 975/1968] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:28.835708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed2a7a-c455-4472-bef1-c5176601840b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:28.837485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-71ed2a7a-c455-4472-bef1-c5176601840b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:28.876059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b478f998-5f84-4fbe-88ca-2669ae15d3b8 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:28.881104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e26f27-8b2a-40cb-bcf7-05fbe8a9d659 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:28.881736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e26f27-8b2a-40cb-bcf7-05fbe8a9d659 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:28.882022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e26f27-8b2a-40cb-bcf7-05fbe8a9d659 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:28.882297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e26f27-8b2a-40cb-bcf7-05fbe8a9d659 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:28.893052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e26f27-8b2a-40cb-bcf7-05fbe8a9d659 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:28.895609 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e26f27-8b2a-40cb-bcf7-05fbe8a9d659 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:28.902584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed2a7a-c455-4472-bef1-c5176601840b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:28.906599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 994/1969] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:28.908739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f138c-7331-4edc-9f0e-6880e8ada74a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:28.909712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f138c-7331-4edc-9f0e-6880e8ada74a service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:28.910048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f138c-7331-4edc-9f0e-6880e8ada74a service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:28.910281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443f138c-7331-4edc-9f0e-6880e8ada74a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:28.933932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443f138c-7331-4edc-9f0e-6880e8ada74a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:28.933932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443f138c-7331-4edc-9f0e-6880e8ada74a service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:28.934669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 976/1970] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:29.419080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e26f27-8b2a-40cb-bcf7-05fbe8a9d659 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:29.420584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-78e26f27-8b2a-40cb-bcf7-05fbe8a9d659 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:29.437684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e26f27-8b2a-40cb-bcf7-05fbe8a9d659 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:29.442501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:29.443956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:29.444336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:29.444667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] PATH_INFO: `/v3/projects/e5f4896d1efd4372a13dbcaeda54bfe2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:29.446656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5f4896d1efd4372a13dbcaeda54bfe2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:29.449262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:29.465414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f138c-7331-4edc-9f0e-6880e8ada74a service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:29.467175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-443f138c-7331-4edc-9f0e-6880e8ada74a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:29.489059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443f138c-7331-4edc-9f0e-6880e8ada74a service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:29.493345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:29.495332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:29.496105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:29.496382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:29.496707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] PATH_INFO: `/v3/projects/ab62f02c8cb14148aaca480aaf75193e` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:29.497970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab62f02c8cb14148aaca480aaf75193e)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:29.500919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:29.516330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5f4896d1efd4372a13dbcaeda54bfe2', 'request_id': 'req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.522864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5f4896d1efd4372a13dbcaeda54bfe2', 'request_id': 'req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.531793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5f4896d1efd4372a13dbcaeda54bfe2', 'request_id': 'req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.543768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] The token cache is being invalidate because project e5f4896d1efd4372a13dbcaeda54bfe2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:29.544070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.544764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.546559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 995/1971] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:29 2022] DELETE /identity/v3/projects/e5f4896d1efd4372a13dbcaeda54bfe2 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:29.551068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:29.551497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab62f02c8cb14148aaca480aaf75193e', 'request_id': 'req-acad372c-f684-4607-abb8-38586a05b425'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.552501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:29.560296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab62f02c8cb14148aaca480aaf75193e', 'request_id': 'req-acad372c-f684-4607-abb8-38586a05b425'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.566727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab62f02c8cb14148aaca480aaf75193e', 'request_id': 'req-acad372c-f684-4607-abb8-38586a05b425'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.578873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] The token cache is being invalidate because project ab62f02c8cb14148aaca480aaf75193e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:29.579164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.579810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.581895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 977/1972] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:29 2022] DELETE /identity/v3/projects/ab62f02c8cb14148aaca480aaf75193e => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:29.585606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:29.586908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:29.622440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c316bd1-99b6-4bd7-98af-4e24f4702f5c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:29.632479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:29.658503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acad372c-f684-4607-abb8-38586a05b425 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:29.664004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:29.664775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:29.665128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:29.665480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] PATH_INFO: `/v3/users/6bb39df7be6b4456a87a99b3bc36fa4d` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:29.666756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] RBAC: Authorizing `identity:delete_user(user_id=6bb39df7be6b4456a87a99b3bc36fa4d)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:29.679937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:29.697428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:29.698600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:29.698997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:29.699418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] PATH_INFO: `/v3/users/5e609c41ba94499285150cb9471e16a7` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:29.702018 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e609c41ba94499285150cb9471e16a7)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:29.704568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:29.740882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6bb39df7be6b4456a87a99b3bc36fa4d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.756350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6bb39df7be6b4456a87a99b3bc36fa4d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.758250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6bb39df7be6b4456a87a99b3bc36fa4d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.760464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6bb39df7be6b4456a87a99b3bc36fa4d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.770368 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e609c41ba94499285150cb9471e16a7'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.770477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6bb39df7be6b4456a87a99b3bc36fa4d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.770828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e609c41ba94499285150cb9471e16a7'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.771544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e609c41ba94499285150cb9471e16a7'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.788245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e609c41ba94499285150cb9471e16a7'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.790071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:29.800554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1f0ebb35-76e8-4bcb-a1f5-5751fbd5d6a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e609c41ba94499285150cb9471e16a7'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:29.805290 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 978/1973] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:29 2022] DELETE /identity/v3/users/6bb39df7be6b4456a87a99b3bc36fa4d => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:29.821700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 996/1974] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:29 2022] DELETE /identity/v3/users/5e609c41ba94499285150cb9471e16a7 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:30.160477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.161967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:30.240531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e6a57e-f1bb-4fac-8d0b-b4ba55a4315d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.306833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.307614 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.308050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.308378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] PATH_INFO: `/v3/projects/54940752312842149aae35037e1f7651` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.309796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] RBAC: Authorizing `identity:delete_project(project_id=54940752312842149aae35037e1f7651)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.312530 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:30.315495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:30.368769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54940752312842149aae35037e1f7651', 'request_id': 'req-f24159fc-6130-4a64-8f20-73d2b64bb877'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:30.376815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54940752312842149aae35037e1f7651', 'request_id': 'req-f24159fc-6130-4a64-8f20-73d2b64bb877'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:30.383998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54940752312842149aae35037e1f7651', 'request_id': 'req-f24159fc-6130-4a64-8f20-73d2b64bb877'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:30.396790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] The token cache is being invalidate because project 54940752312842149aae35037e1f7651 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:30.397211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:30.397967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:30.399464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 979/1975] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:30 2022] DELETE /identity/v3/projects/54940752312842149aae35037e1f7651 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:30.405136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b48299b-a518-44cf-aa9a-9bd95a5b1325 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.405430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b48299b-a518-44cf-aa9a-9bd95a5b1325 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.406166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b48299b-a518-44cf-aa9a-9bd95a5b1325 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.427342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-0b48299b-a518-44cf-aa9a-9bd95a5b1325 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:30.429060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0b48299b-a518-44cf-aa9a-9bd95a5b1325 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:30.474850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.476336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:30.502095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 997/1976] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:30.511370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab46e4bd-d7fa-4076-b353-3bf017cc0596 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.511665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab46e4bd-d7fa-4076-b353-3bf017cc0596 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.511948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab46e4bd-d7fa-4076-b353-3bf017cc0596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.534703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-ab46e4bd-d7fa-4076-b353-3bf017cc0596 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:30.535935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ab46e4bd-d7fa-4076-b353-3bf017cc0596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:30.544326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 998/1977] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:30.545498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24159fc-6130-4a64-8f20-73d2b64bb877 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.548696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab46e4bd-d7fa-4076-b353-3bf017cc0596 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.552114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.552848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.553125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.553402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] PATH_INFO: `/v3/projects/5d3a871b601341a4a4d3d3b5bbd27b5b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.554626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d3a871b601341a4a4d3d3b5bbd27b5b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.556547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:30.558279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab46e4bd-d7fa-4076-b353-3bf017cc0596 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.563830 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abb90fd-c892-468e-aa35-c43d26690dcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.564511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abb90fd-c892-468e-aa35-c43d26690dcc admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.564833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abb90fd-c892-468e-aa35-c43d26690dcc admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.566287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abb90fd-c892-468e-aa35-c43d26690dcc admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.567008 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abb90fd-c892-468e-aa35-c43d26690dcc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:30.567323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abb90fd-c892-468e-aa35-c43d26690dcc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.568986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abb90fd-c892-468e-aa35-c43d26690dcc admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:30.585242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 999/1978] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:48:30.586702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abb90fd-c892-468e-aa35-c43d26690dcc admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.596737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abb90fd-c892-468e-aa35-c43d26690dcc admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.606692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d3a871b601341a4a4d3d3b5bbd27b5b', 'request_id': 'req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:30.614164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d3a871b601341a4a4d3d3b5bbd27b5b', 'request_id': 'req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:30.619916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d3a871b601341a4a4d3d3b5bbd27b5b', 'request_id': 'req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:30.634101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] The token cache is being invalidate because project 5d3a871b601341a4a4d3d3b5bbd27b5b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:30.634646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:30.635289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b780b149-4d3c-4a47-a7a3-9d5197b4fec0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:30.637222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 980/1979] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:30 2022] DELETE /identity/v3/projects/5d3a871b601341a4a4d3d3b5bbd27b5b => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:30.642357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc134ed8-bcd2-49ba-b7d2-300a2ffa5bf0 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.642621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc134ed8-bcd2-49ba-b7d2-300a2ffa5bf0 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.642909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc134ed8-bcd2-49ba-b7d2-300a2ffa5bf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.670122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aeb154-167b-4495-9b73-2a851ece1067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.670975 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-fc134ed8-bcd2-49ba-b7d2-300a2ffa5bf0 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:30.671085 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aeb154-167b-4495-9b73-2a851ece1067 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.671369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aeb154-167b-4495-9b73-2a851ece1067 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.671643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aeb154-167b-4495-9b73-2a851ece1067 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.672202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fc134ed8-bcd2-49ba-b7d2-300a2ffa5bf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:30.672316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13aeb154-167b-4495-9b73-2a851ece1067 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.674454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13aeb154-167b-4495-9b73-2a851ece1067 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:30.680630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 981/1980] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:30.688435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9710a186-8f93-4a3f-a7df-5fb4e1c89fd5 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.688869 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9710a186-8f93-4a3f-a7df-5fb4e1c89fd5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.689158 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9710a186-8f93-4a3f-a7df-5fb4e1c89fd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.711310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1000/1981] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:30 2022] POST /identity/v3/projects => generated 409 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:30.711526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-9710a186-8f93-4a3f-a7df-5fb4e1c89fd5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:30.711526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9710a186-8f93-4a3f-a7df-5fb4e1c89fd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:30.712395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aeb154-167b-4495-9b73-2a851ece1067 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.714095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-13aeb154-167b-4495-9b73-2a851ece1067 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:30.720797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:30.766248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:30.774728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 982/1982] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:30.778919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9710a186-8f93-4a3f-a7df-5fb4e1c89fd5 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.786146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aeb154-167b-4495-9b73-2a851ece1067 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.788915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9710a186-8f93-4a3f-a7df-5fb4e1c89fd5 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.791090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610c6652-f5c9-4b6d-8f66-324080c8f05f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.791722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610c6652-f5c9-4b6d-8f66-324080c8f05f admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.792013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610c6652-f5c9-4b6d-8f66-324080c8f05f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.792296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610c6652-f5c9-4b6d-8f66-324080c8f05f admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.793030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610c6652-f5c9-4b6d-8f66-324080c8f05f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.793510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7935ddbf-91ea-4fad-8837-d9d9eb611111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.794122 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935ddbf-91ea-4fad-8837-d9d9eb611111 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.794410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935ddbf-91ea-4fad-8837-d9d9eb611111 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.794694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7935ddbf-91ea-4fad-8837-d9d9eb611111 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.794965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610c6652-f5c9-4b6d-8f66-324080c8f05f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:30.795397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7935ddbf-91ea-4fad-8837-d9d9eb611111 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:30.795991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7935ddbf-91ea-4fad-8837-d9d9eb611111 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.797611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7935ddbf-91ea-4fad-8837-d9d9eb611111 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:30.810224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 983/1983] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:48:30.814658 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7935ddbf-91ea-4fad-8837-d9d9eb611111 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.820676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1001/1984] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:30 2022] POST /identity/v3/users => generated 305 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:30.822355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610c6652-f5c9-4b6d-8f66-324080c8f05f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.824542 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7935ddbf-91ea-4fad-8837-d9d9eb611111 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.829466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bf8320-8394-4f44-be97-bb781e5e3010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.830096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bf8320-8394-4f44-be97-bb781e5e3010 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.830412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bf8320-8394-4f44-be97-bb781e5e3010 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.831254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bf8320-8394-4f44-be97-bb781e5e3010 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.831918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bf8320-8394-4f44-be97-bb781e5e3010 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.832641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610c6652-f5c9-4b6d-8f66-324080c8f05f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.833856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bf8320-8394-4f44-be97-bb781e5e3010 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:30.837305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860cf913-8db0-4373-b9b1-8606be320a20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.837940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860cf913-8db0-4373-b9b1-8606be320a20 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.838224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860cf913-8db0-4373-b9b1-8606be320a20 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.838503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860cf913-8db0-4373-b9b1-8606be320a20 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.839194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860cf913-8db0-4373-b9b1-8606be320a20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.840798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860cf913-8db0-4373-b9b1-8606be320a20 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:30.851699 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1002/1985] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:30.856238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860cf913-8db0-4373-b9b1-8606be320a20 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.862757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 984/1986] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:30 2022] POST /identity/v3/projects => generated 379 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:30.866331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860cf913-8db0-4373-b9b1-8606be320a20 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.866458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bf8320-8394-4f44-be97-bb781e5e3010 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.930775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f627e2af-ec4e-4107-8af8-27d395ab9c39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.931465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f627e2af-ec4e-4107-8af8-27d395ab9c39 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.931761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f627e2af-ec4e-4107-8af8-27d395ab9c39 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.932040 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f627e2af-ec4e-4107-8af8-27d395ab9c39 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.932787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f627e2af-ec4e-4107-8af8-27d395ab9c39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.934394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f627e2af-ec4e-4107-8af8-27d395ab9c39 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:30.942351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bf8320-8394-4f44-be97-bb781e5e3010 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.947178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bf2a3c-811c-436a-aa88-ee199cb397ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.948649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1003/1987] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:30 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:30.948783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bf2a3c-811c-436a-aa88-ee199cb397ae admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.948890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bf2a3c-811c-436a-aa88-ee199cb397ae admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.949170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84bf2a3c-811c-436a-aa88-ee199cb397ae admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.949840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bf2a3c-811c-436a-aa88-ee199cb397ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.951995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84bf2a3c-811c-436a-aa88-ee199cb397ae admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:30.954516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f627e2af-ec4e-4107-8af8-27d395ab9c39 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.967148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f627e2af-ec4e-4107-8af8-27d395ab9c39 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:30.974764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6524f0b-13e7-4705-bc0c-7d22be003271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:30.974764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6524f0b-13e7-4705-bc0c-7d22be003271 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:30.975065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6524f0b-13e7-4705-bc0c-7d22be003271 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:30.975643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6524f0b-13e7-4705-bc0c-7d22be003271 admin admin] PATH_INFO: `/v3/projects/edfc7b2ca46145f5b962fe4435b0d591/users/2877a2292fc94370bf07a4f2e1e470cb/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:30.986310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 985/1988] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:30 2022] POST /identity/v3/users => generated 290 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:30.990577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bf2a3c-811c-436a-aa88-ee199cb397ae admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:30.994277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6524f0b-13e7-4705-bc0c-7d22be003271 admin admin] RBAC: Authorizing `identity:create_grant(project_id=edfc7b2ca46145f5b962fe4435b0d591, user_id=2877a2292fc94370bf07a4f2e1e470cb, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:30.997336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6524f0b-13e7-4705-bc0c-7d22be003271 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.006546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84bf2a3c-811c-436a-aa88-ee199cb397ae admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.023113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1004/1989] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:30 2022] PUT /identity/v3/projects/edfc7b2ca46145f5b962fe4435b0d591/users/2877a2292fc94370bf07a4f2e1e470cb/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:31.029244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b768729-018f-4bed-a931-92b5104089f4 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.029875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b768729-018f-4bed-a931-92b5104089f4 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.030496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b768729-018f-4bed-a931-92b5104089f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.080310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-3b768729-018f-4bed-a931-92b5104089f4 None None] MFA Rules not processed for user `2877a2292fc94370bf07a4f2e1e470cb`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:31.082117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3b768729-018f-4bed-a931-92b5104089f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:31.098059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fecdad9-6be6-4a58-b28b-ad26a2577567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.098726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fecdad9-6be6-4a58-b28b-ad26a2577567 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.099427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fecdad9-6be6-4a58-b28b-ad26a2577567 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.099713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fecdad9-6be6-4a58-b28b-ad26a2577567 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.100424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fecdad9-6be6-4a58-b28b-ad26a2577567 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.102327 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fecdad9-6be6-4a58-b28b-ad26a2577567 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.116227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 986/1990] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:30 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:31.117650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fecdad9-6be6-4a58-b28b-ad26a2577567 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.127257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fecdad9-6be6-4a58-b28b-ad26a2577567 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.132178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605b9e52-cde1-4701-b123-159aaaaf6d59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.132812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605b9e52-cde1-4701-b123-159aaaaf6d59 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.133100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605b9e52-cde1-4701-b123-159aaaaf6d59 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.133379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605b9e52-cde1-4701-b123-159aaaaf6d59 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.134070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605b9e52-cde1-4701-b123-159aaaaf6d59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.135667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605b9e52-cde1-4701-b123-159aaaaf6d59 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.137041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:31.148646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 987/1991] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:31.152318 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605b9e52-cde1-4701-b123-159aaaaf6d59 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.162654 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605b9e52-cde1-4701-b123-159aaaaf6d59 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.162804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1005/1992] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:31 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:31.167000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b768729-018f-4bed-a931-92b5104089f4 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.167545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a59c4a-47b1-4a72-b1ed-1a0bbebb19e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.168205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a59c4a-47b1-4a72-b1ed-1a0bbebb19e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.168492 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a59c4a-47b1-4a72-b1ed-1a0bbebb19e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.168801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a59c4a-47b1-4a72-b1ed-1a0bbebb19e9 admin admin] PATH_INFO: `/v3/projects/03046bc3e7ec4b50b15e0479d1da89d2/users/35bd7418b6f24c30b2b7e2aadf11aa85/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.178938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b768729-018f-4bed-a931-92b5104089f4 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.184274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a59c4a-47b1-4a72-b1ed-1a0bbebb19e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=03046bc3e7ec4b50b15e0479d1da89d2, user_id=35bd7418b6f24c30b2b7e2aadf11aa85, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.185339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ed2c82-e77e-4507-8b53-e16506d123ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.185973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ed2c82-e77e-4507-8b53-e16506d123ed admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.186249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ed2c82-e77e-4507-8b53-e16506d123ed admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.186530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ed2c82-e77e-4507-8b53-e16506d123ed admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.186988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98a59c4a-47b1-4a72-b1ed-1a0bbebb19e9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.187231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ed2c82-e77e-4507-8b53-e16506d123ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.189382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7ed2c82-e77e-4507-8b53-e16506d123ed admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.202621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 988/1993] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:31 2022] PUT /identity/v3/projects/03046bc3e7ec4b50b15e0479d1da89d2/users/35bd7418b6f24c30b2b7e2aadf11aa85/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:31.214005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5085f213-8980-484c-98a4-f67842969709 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.214509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5085f213-8980-484c-98a4-f67842969709 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.214812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5085f213-8980-484c-98a4-f67842969709 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.222372 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1006/1994] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:31 2022] POST /identity/v3/projects => generated 405 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:31.226150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ed2c82-e77e-4507-8b53-e16506d123ed admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.243875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:31.250141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-5085f213-8980-484c-98a4-f67842969709 None None] MFA Rules not processed for user `35bd7418b6f24c30b2b7e2aadf11aa85`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:31.254646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5085f213-8980-484c-98a4-f67842969709 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:31.322696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7ed2c82-e77e-4507-8b53-e16506d123ed admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.327751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cdc06-c02c-4362-8337-36a37c53356c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.328468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cdc06-c02c-4362-8337-36a37c53356c admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.328872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cdc06-c02c-4362-8337-36a37c53356c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.329226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8cdc06-c02c-4362-8337-36a37c53356c admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.329972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8cdc06-c02c-4362-8337-36a37c53356c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.332157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8cdc06-c02c-4362-8337-36a37c53356c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.340140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 989/1995] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:31 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:31.344479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5085f213-8980-484c-98a4-f67842969709 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.356759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1007/1996] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:31 2022] POST /identity/v3/users => generated 309 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:31.360356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5085f213-8980-484c-98a4-f67842969709 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.361013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cdc06-c02c-4362-8337-36a37c53356c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.367052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0586e59-e320-4a5d-8df5-f9648e55d586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.367691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0586e59-e320-4a5d-8df5-f9648e55d586 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.367978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0586e59-e320-4a5d-8df5-f9648e55d586 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.368260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0586e59-e320-4a5d-8df5-f9648e55d586 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.369169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0586e59-e320-4a5d-8df5-f9648e55d586 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.371193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0586e59-e320-4a5d-8df5-f9648e55d586 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.375638 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8cdc06-c02c-4362-8337-36a37c53356c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.381147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d0da9e-0d1c-4f73-9dfc-6bd5bdf3f356 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.381792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d0da9e-0d1c-4f73-9dfc-6bd5bdf3f356 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.382083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d0da9e-0d1c-4f73-9dfc-6bd5bdf3f356 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.382362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d0da9e-0d1c-4f73-9dfc-6bd5bdf3f356 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.383496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d0da9e-0d1c-4f73-9dfc-6bd5bdf3f356 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.385560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d0da9e-0d1c-4f73-9dfc-6bd5bdf3f356 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.398524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1008/1997] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:31.402782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d0da9e-0d1c-4f73-9dfc-6bd5bdf3f356 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.408133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 990/1998] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:31 2022] POST /identity/v3/projects => generated 379 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:31.410694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0586e59-e320-4a5d-8df5-f9648e55d586 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.414231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d0da9e-0d1c-4f73-9dfc-6bd5bdf3f356 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.481340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0586e59-e320-4a5d-8df5-f9648e55d586 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.486009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dcb0f3-7013-4509-850d-4702c073f089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.486624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dcb0f3-7013-4509-850d-4702c073f089 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.487480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dcb0f3-7013-4509-850d-4702c073f089 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.487859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dcb0f3-7013-4509-850d-4702c073f089 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.488792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dcb0f3-7013-4509-850d-4702c073f089 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.490818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dcb0f3-7013-4509-850d-4702c073f089 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.492173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb652c43-8343-49da-b52c-40da3134cf40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.492971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb652c43-8343-49da-b52c-40da3134cf40 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.493407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb652c43-8343-49da-b52c-40da3134cf40 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.493792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb652c43-8343-49da-b52c-40da3134cf40 admin admin] PATH_INFO: `/v3/projects/f5ab7676a7894722904c2c2958d1ee5e/users/b539ec621f64444f9b2cfe5e40b45dce/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.510820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb652c43-8343-49da-b52c-40da3134cf40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5ab7676a7894722904c2c2958d1ee5e, user_id=b539ec621f64444f9b2cfe5e40b45dce, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.513475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb652c43-8343-49da-b52c-40da3134cf40 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.520729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 991/1999] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:31 2022] POST /identity/v3/users => generated 296 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:31.524219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dcb0f3-7013-4509-850d-4702c073f089 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.531033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1009/2000] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:31 2022] PUT /identity/v3/projects/f5ab7676a7894722904c2c2958d1ee5e/users/b539ec621f64444f9b2cfe5e40b45dce/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:31.534507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb652c43-8343-49da-b52c-40da3134cf40 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.537070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dcb0f3-7013-4509-850d-4702c073f089 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.568175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:31.607953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e625ff-789f-4ac6-99ba-4e90b1290e6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.608746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e625ff-789f-4ac6-99ba-4e90b1290e6a admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.609096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e625ff-789f-4ac6-99ba-4e90b1290e6a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.609441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e625ff-789f-4ac6-99ba-4e90b1290e6a admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.610293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e625ff-789f-4ac6-99ba-4e90b1290e6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.612635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e625ff-789f-4ac6-99ba-4e90b1290e6a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.617389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb652c43-8343-49da-b52c-40da3134cf40 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.624103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a208459e-7d8f-4489-a603-588499c364b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.624860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a208459e-7d8f-4489-a603-588499c364b2 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.625277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a208459e-7d8f-4489-a603-588499c364b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.625825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 992/2001] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:31 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:31.626280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a208459e-7d8f-4489-a603-588499c364b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.627194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a208459e-7d8f-4489-a603-588499c364b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.629259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a208459e-7d8f-4489-a603-588499c364b2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.631243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e625ff-789f-4ac6-99ba-4e90b1290e6a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.642398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e625ff-789f-4ac6-99ba-4e90b1290e6a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.643556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1010/2002] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:31 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:31.656286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de25a1a-64ad-49fc-9d52-5f3135f7e9aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.657392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de25a1a-64ad-49fc-9d52-5f3135f7e9aa admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.657668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a208459e-7d8f-4489-a603-588499c364b2 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.657793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de25a1a-64ad-49fc-9d52-5f3135f7e9aa admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.658137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de25a1a-64ad-49fc-9d52-5f3135f7e9aa admin admin] PATH_INFO: `/v3/projects/15dcab582c2c405bbc665085700277d9/users/f78412ae0dab499d9bd52431a738aa19/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.673429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:31.691384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a208459e-7d8f-4489-a603-588499c364b2 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.693347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de25a1a-64ad-49fc-9d52-5f3135f7e9aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=15dcab582c2c405bbc665085700277d9, user_id=f78412ae0dab499d9bd52431a738aa19, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.696437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de25a1a-64ad-49fc-9d52-5f3135f7e9aa admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.710718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c5dae1-dde3-48be-94db-d8ef6d35a4c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.711616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c5dae1-dde3-48be-94db-d8ef6d35a4c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.712051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c5dae1-dde3-48be-94db-d8ef6d35a4c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.712482 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c5dae1-dde3-48be-94db-d8ef6d35a4c2 admin admin] PATH_INFO: `/v3/projects/f5ab7676a7894722904c2c2958d1ee5e/users/b539ec621f64444f9b2cfe5e40b45dce/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.715340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c5dae1-dde3-48be-94db-d8ef6d35a4c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5ab7676a7894722904c2c2958d1ee5e, user_id=b539ec621f64444f9b2cfe5e40b45dce, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.715534 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 993/2003] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:31 2022] PUT /identity/v3/projects/15dcab582c2c405bbc665085700277d9/users/f78412ae0dab499d9bd52431a738aa19/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:31.718811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c5dae1-dde3-48be-94db-d8ef6d35a4c2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.719440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de25a1a-64ad-49fc-9d52-5f3135f7e9aa admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.730601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1011/2004] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:31 2022] PUT /identity/v3/projects/f5ab7676a7894722904c2c2958d1ee5e/users/b539ec621f64444f9b2cfe5e40b45dce/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:31.741470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a8483f-ff38-4a04-b19b-13162efd6c32 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.741696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a8483f-ff38-4a04-b19b-13162efd6c32 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.741976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a8483f-ff38-4a04-b19b-13162efd6c32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.771506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d0a8483f-ff38-4a04-b19b-13162efd6c32 None None] MFA Rules not processed for user `b539ec621f64444f9b2cfe5e40b45dce`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:31.772782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d0a8483f-ff38-4a04-b19b-13162efd6c32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:31.783172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de25a1a-64ad-49fc-9d52-5f3135f7e9aa admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.843517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23b0296-c9ed-4c57-a415-625a82a489d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.844157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23b0296-c9ed-4c57-a415-625a82a489d0 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.844443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23b0296-c9ed-4c57-a415-625a82a489d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.844761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23b0296-c9ed-4c57-a415-625a82a489d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.845462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23b0296-c9ed-4c57-a415-625a82a489d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.847645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23b0296-c9ed-4c57-a415-625a82a489d0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.852058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1012/2005] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:31 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:31.858488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 994/2006] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:31 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:31.862633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a8483f-ff38-4a04-b19b-13162efd6c32 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.863897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23b0296-c9ed-4c57-a415-625a82a489d0 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:31.865816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c23b0296-c9ed-4c57-a415-625a82a489d0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:31.872950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a8483f-ff38-4a04-b19b-13162efd6c32 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.878694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109dbf1c-ae8c-4e43-9ed5-1e47d8331491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.879458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109dbf1c-ae8c-4e43-9ed5-1e47d8331491 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.879782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109dbf1c-ae8c-4e43-9ed5-1e47d8331491 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.880061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109dbf1c-ae8c-4e43-9ed5-1e47d8331491 admin admin] PATH_INFO: `/v3/projects/15dcab582c2c405bbc665085700277d9/users/f78412ae0dab499d9bd52431a738aa19/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.882156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109dbf1c-ae8c-4e43-9ed5-1e47d8331491 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15dcab582c2c405bbc665085700277d9, user_id=f78412ae0dab499d9bd52431a738aa19, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:31.885022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109dbf1c-ae8c-4e43-9ed5-1e47d8331491 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:31.896735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1013/2007] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:31 2022] PUT /identity/v3/projects/15dcab582c2c405bbc665085700277d9/users/f78412ae0dab499d9bd52431a738aa19/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:31.906814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f26d50-8a6f-41d2-a06a-b611bc667900 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.907116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f26d50-8a6f-41d2-a06a-b611bc667900 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.907401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f26d50-8a6f-41d2-a06a-b611bc667900 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:31.920120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:31.932804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23b0296-c9ed-4c57-a415-625a82a489d0 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:31.937850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-75f26d50-8a6f-41d2-a06a-b611bc667900 None None] MFA Rules not processed for user `f78412ae0dab499d9bd52431a738aa19`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:31.939084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-75f26d50-8a6f-41d2-a06a-b611bc667900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:31.983704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549701f4-c59b-4012-8db9-1d95dc135e17 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:31.984336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549701f4-c59b-4012-8db9-1d95dc135e17 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:31.984643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549701f4-c59b-4012-8db9-1d95dc135e17 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:31.984928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549701f4-c59b-4012-8db9-1d95dc135e17 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:32.009517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1014/2008] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:31 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:32.021959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f26d50-8a6f-41d2-a06a-b611bc667900 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:32.023355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-75f26d50-8a6f-41d2-a06a-b611bc667900 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:32.035035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f26d50-8a6f-41d2-a06a-b611bc667900 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:32.038652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea76dd7b-05fa-40c0-b267-96427ad8969c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:32.039259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea76dd7b-05fa-40c0-b267-96427ad8969c service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:32.039541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea76dd7b-05fa-40c0-b267-96427ad8969c service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:32.039814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea76dd7b-05fa-40c0-b267-96427ad8969c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:32.048065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549701f4-c59b-4012-8db9-1d95dc135e17 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:32.048927 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea76dd7b-05fa-40c0-b267-96427ad8969c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:32.049806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549701f4-c59b-4012-8db9-1d95dc135e17 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:32.050509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea76dd7b-05fa-40c0-b267-96427ad8969c service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:32.059962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1015/2009] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:32 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:32.061564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 995/2010] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:31 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:32.147475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea76dd7b-05fa-40c0-b267-96427ad8969c service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:32.157341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea76dd7b-05fa-40c0-b267-96427ad8969c service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:32.163725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5f0a5-0f4a-4d68-8df5-b651c5136d9f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:32.164666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5f0a5-0f4a-4d68-8df5-b651c5136d9f service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:32.165155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5f0a5-0f4a-4d68-8df5-b651c5136d9f service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:32.165350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed5f0a5-0f4a-4d68-8df5-b651c5136d9f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:32.218088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:32.251847 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed5f0a5-0f4a-4d68-8df5-b651c5136d9f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:32.255227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed5f0a5-0f4a-4d68-8df5-b651c5136d9f service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:32.271424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1016/2011] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:32 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:32.273552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549701f4-c59b-4012-8db9-1d95dc135e17 service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:32.302261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549701f4-c59b-4012-8db9-1d95dc135e17 service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:32.306463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e2a4db-2b5c-4d68-a7f8-6e06b6ac9d1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:32.307249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e2a4db-2b5c-4d68-a7f8-6e06b6ac9d1b service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:32.307815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e2a4db-2b5c-4d68-a7f8-6e06b6ac9d1b service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:32.308234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e2a4db-2b5c-4d68-a7f8-6e06b6ac9d1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:32.387639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e2a4db-2b5c-4d68-a7f8-6e06b6ac9d1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:32.389280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e2a4db-2b5c-4d68-a7f8-6e06b6ac9d1b service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:32.404983 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 996/2012] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:32 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:42.266455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=12 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:42.398478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=78 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:46.286950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5f0a5-0f4a-4d68-8df5-b651c5136d9f service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:46.360522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed5f0a5-0f4a-4d68-8df5-b651c5136d9f service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:46.365753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:46.366988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:46.366988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:46.366988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] PATH_INFO: `/v3/users/2877a2292fc94370bf07a4f2e1e470cb` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:46.368127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=2877a2292fc94370bf07a4f2e1e470cb)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:46.370088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:46.430396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2877a2292fc94370bf07a4f2e1e470cb'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:46.431065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2877a2292fc94370bf07a4f2e1e470cb'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:46.431684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2877a2292fc94370bf07a4f2e1e470cb'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:46.448155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2877a2292fc94370bf07a4f2e1e470cb'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:46.456399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2877a2292fc94370bf07a4f2e1e470cb'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:46.483936 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1017/2013] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:46 2022] DELETE /identity/v3/users/2877a2292fc94370bf07a4f2e1e470cb => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:46.489769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b981d3c3-d441-4f60-8b22-acbdeb5dc59f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:46.490272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b981d3c3-d441-4f60-8b22-acbdeb5dc59f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:46.490383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b981d3c3-d441-4f60-8b22-acbdeb5dc59f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:46.509669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-b981d3c3-d441-4f60-8b22-acbdeb5dc59f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:46.510898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b981d3c3-d441-4f60-8b22-acbdeb5dc59f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:46.587464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 997/2014] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:46.602591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:46.610592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:46.698157 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b316822-6204-4bf1-826f-e8db6d2c0a1e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:46.704387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea17991-6555-4991-bc1a-890d03eedeaa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:46.705859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea17991-6555-4991-bc1a-890d03eedeaa service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:46.706199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea17991-6555-4991-bc1a-890d03eedeaa service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:46.706543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea17991-6555-4991-bc1a-890d03eedeaa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:46.716657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea17991-6555-4991-bc1a-890d03eedeaa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:46.718109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea17991-6555-4991-bc1a-890d03eedeaa service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:46.730079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1018/2015] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:47.131897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b981d3c3-d441-4f60-8b22-acbdeb5dc59f None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:47.133312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b981d3c3-d441-4f60-8b22-acbdeb5dc59f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:47.153353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b981d3c3-d441-4f60-8b22-acbdeb5dc59f None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:47.158793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:47.159617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:47.159912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:47.160194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] PATH_INFO: `/v3/projects/edfc7b2ca46145f5b962fe4435b0d591` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:47.161438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] RBAC: Authorizing `identity:delete_project(project_id=edfc7b2ca46145f5b962fe4435b0d591)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:47.163430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:47.215956 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'edfc7b2ca46145f5b962fe4435b0d591', 'request_id': 'req-15b85883-3fcb-4df6-aa86-feac0225d253'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.224427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'edfc7b2ca46145f5b962fe4435b0d591', 'request_id': 'req-15b85883-3fcb-4df6-aa86-feac0225d253'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.230929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'edfc7b2ca46145f5b962fe4435b0d591', 'request_id': 'req-15b85883-3fcb-4df6-aa86-feac0225d253'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.243133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] The token cache is being invalidate because project edfc7b2ca46145f5b962fe4435b0d591 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:47.243429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.244083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.245628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 998/2016] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:47 2022] DELETE /identity/v3/projects/edfc7b2ca46145f5b962fe4435b0d591 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:47.249474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea17991-6555-4991-bc1a-890d03eedeaa service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:47.259455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea17991-6555-4991-bc1a-890d03eedeaa service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:47.278232 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:47.320434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:47.321135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:47.321392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:47.321680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] PATH_INFO: `/v3/users/b539ec621f64444f9b2cfe5e40b45dce` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:47.322852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b539ec621f64444f9b2cfe5e40b45dce)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:47.324809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:47.371435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b539ec621f64444f9b2cfe5e40b45dce'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.372437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b539ec621f64444f9b2cfe5e40b45dce'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.373083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b539ec621f64444f9b2cfe5e40b45dce'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.385667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b539ec621f64444f9b2cfe5e40b45dce'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.393486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-5f940f07-8632-4321-8166-b7075b412126 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b539ec621f64444f9b2cfe5e40b45dce'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.407228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1019/2017] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:47 2022] DELETE /identity/v3/users/b539ec621f64444f9b2cfe5e40b45dce => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:47.726227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:47.727692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:47.743554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b85883-3fcb-4df6-aa86-feac0225d253 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:47.811851 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:47.812519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:47.812846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:47.813147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] PATH_INFO: `/v3/projects/f5ab7676a7894722904c2c2958d1ee5e` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:47.815105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5ab7676a7894722904c2c2958d1ee5e)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:47.817126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:47.847941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:47.865880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5ab7676a7894722904c2c2958d1ee5e', 'request_id': 'req-ad77797a-6d1c-4c76-93cf-07d9e5bba694'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.872762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5ab7676a7894722904c2c2958d1ee5e', 'request_id': 'req-ad77797a-6d1c-4c76-93cf-07d9e5bba694'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.878426 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5ab7676a7894722904c2c2958d1ee5e', 'request_id': 'req-ad77797a-6d1c-4c76-93cf-07d9e5bba694'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.889668 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] The token cache is being invalidate because project f5ab7676a7894722904c2c2958d1ee5e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:47.889957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.890598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ad77797a-6d1c-4c76-93cf-07d9e5bba694 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:47.892094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 999/2018] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:47 2022] DELETE /identity/v3/projects/f5ab7676a7894722904c2c2958d1ee5e => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:47.898410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a51b6fb-473a-4e37-af30-346e7f12274b None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:47.898642 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a51b6fb-473a-4e37-af30-346e7f12274b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:47.898924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a51b6fb-473a-4e37-af30-346e7f12274b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:47.918821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-8a51b6fb-473a-4e37-af30-346e7f12274b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:47.920031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8a51b6fb-473a-4e37-af30-346e7f12274b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:47.977824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1020/2019] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:47.985449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb90378-4017-4722-af99-8f60f9dd3fbd None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:47.985742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb90378-4017-4722-af99-8f60f9dd3fbd None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:47.986154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb90378-4017-4722-af99-8f60f9dd3fbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.006217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-7eb90378-4017-4722-af99-8f60f9dd3fbd None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:48.007431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7eb90378-4017-4722-af99-8f60f9dd3fbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:48.015367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1000/2020] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:48.020099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a51b6fb-473a-4e37-af30-346e7f12274b None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.029719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a51b6fb-473a-4e37-af30-346e7f12274b None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.035000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884dcec8-70e6-4d03-b2ab-5dc8222c78fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.035619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884dcec8-70e6-4d03-b2ab-5dc8222c78fe admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.035905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884dcec8-70e6-4d03-b2ab-5dc8222c78fe admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.036181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884dcec8-70e6-4d03-b2ab-5dc8222c78fe admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.036947 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884dcec8-70e6-4d03-b2ab-5dc8222c78fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:48.037257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884dcec8-70e6-4d03-b2ab-5dc8222c78fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.038853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884dcec8-70e6-4d03-b2ab-5dc8222c78fe admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.051454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1021/2021] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:48:48.055763 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb90378-4017-4722-af99-8f60f9dd3fbd None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.065495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb90378-4017-4722-af99-8f60f9dd3fbd None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.070219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de45f73a-17f3-4ba2-8029-255a9ec8db83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.070830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de45f73a-17f3-4ba2-8029-255a9ec8db83 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.071103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de45f73a-17f3-4ba2-8029-255a9ec8db83 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.071392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de45f73a-17f3-4ba2-8029-255a9ec8db83 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.072068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de45f73a-17f3-4ba2-8029-255a9ec8db83 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.073990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de45f73a-17f3-4ba2-8029-255a9ec8db83 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.102132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1001/2022] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:48 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:48.106234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884dcec8-70e6-4d03-b2ab-5dc8222c78fe admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.116755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884dcec8-70e6-4d03-b2ab-5dc8222c78fe admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.143158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:48.183487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c7f338-7b40-4ecb-a661-3c3998b0ad6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.184119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c7f338-7b40-4ecb-a661-3c3998b0ad6c admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.184406 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c7f338-7b40-4ecb-a661-3c3998b0ad6c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.184714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c7f338-7b40-4ecb-a661-3c3998b0ad6c admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.185390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c7f338-7b40-4ecb-a661-3c3998b0ad6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.187330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c7f338-7b40-4ecb-a661-3c3998b0ad6c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.216692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1022/2023] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:48 2022] POST /identity/v3/users => generated 293 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:48.220039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de45f73a-17f3-4ba2-8029-255a9ec8db83 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.231967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de45f73a-17f3-4ba2-8029-255a9ec8db83 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.237307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd900a8-d520-479d-926d-6ff592d9687c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.238062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd900a8-d520-479d-926d-6ff592d9687c admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.238456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd900a8-d520-479d-926d-6ff592d9687c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.238843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd900a8-d520-479d-926d-6ff592d9687c admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.240059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd900a8-d520-479d-926d-6ff592d9687c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.241420 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd900a8-d520-479d-926d-6ff592d9687c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.252792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1002/2024] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:48.256827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c7f338-7b40-4ecb-a661-3c3998b0ad6c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.268896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c7f338-7b40-4ecb-a661-3c3998b0ad6c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.274214 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131df187-a3e9-4dbb-a0a7-bed3ad7bb89c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.275218 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131df187-a3e9-4dbb-a0a7-bed3ad7bb89c admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.275433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131df187-a3e9-4dbb-a0a7-bed3ad7bb89c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.275712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131df187-a3e9-4dbb-a0a7-bed3ad7bb89c admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.276494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131df187-a3e9-4dbb-a0a7-bed3ad7bb89c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.281110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131df187-a3e9-4dbb-a0a7-bed3ad7bb89c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.290429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1023/2025] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:48 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:48.294572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd900a8-d520-479d-926d-6ff592d9687c admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.307441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd900a8-d520-479d-926d-6ff592d9687c admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.313232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d7594a-d4d9-47b0-85fa-ab55782208ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.314084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d7594a-d4d9-47b0-85fa-ab55782208ab admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.314532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d7594a-d4d9-47b0-85fa-ab55782208ab admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.314794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d7594a-d4d9-47b0-85fa-ab55782208ab admin admin] PATH_INFO: `/v3/projects/4093a675559440daa934166089660e0e/users/d7b042c54b5143519ef94543256fb57b/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.330218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d7594a-d4d9-47b0-85fa-ab55782208ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=4093a675559440daa934166089660e0e, user_id=d7b042c54b5143519ef94543256fb57b, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.333009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d7594a-d4d9-47b0-85fa-ab55782208ab admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.346641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1003/2026] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:48 2022] PUT /identity/v3/projects/4093a675559440daa934166089660e0e/users/d7b042c54b5143519ef94543256fb57b/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:48.356820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359158e7-4b90-427b-a34c-f7410f9ae9f8 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.357246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359158e7-4b90-427b-a34c-f7410f9ae9f8 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.357432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359158e7-4b90-427b-a34c-f7410f9ae9f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.393687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-359158e7-4b90-427b-a34c-f7410f9ae9f8 None None] MFA Rules not processed for user `d7b042c54b5143519ef94543256fb57b`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:48.395063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-359158e7-4b90-427b-a34c-f7410f9ae9f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:48.463038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1024/2027] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:48.467584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d7594a-d4d9-47b0-85fa-ab55782208ab admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.535920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d7594a-d4d9-47b0-85fa-ab55782208ab admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.541171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfcacc0-d871-42bb-aae4-141d7cd7b860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.541795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfcacc0-d871-42bb-aae4-141d7cd7b860 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.542076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfcacc0-d871-42bb-aae4-141d7cd7b860 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.542351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfcacc0-d871-42bb-aae4-141d7cd7b860 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.543026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfcacc0-d871-42bb-aae4-141d7cd7b860 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.544991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfcacc0-d871-42bb-aae4-141d7cd7b860 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.572112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1004/2028] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:48 2022] POST /identity/v3/projects => generated 385 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:48.576334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359158e7-4b90-427b-a34c-f7410f9ae9f8 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.587484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359158e7-4b90-427b-a34c-f7410f9ae9f8 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.643326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2765bb1f-8b43-4b1b-a9c9-65595f4cea32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.643949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2765bb1f-8b43-4b1b-a9c9-65595f4cea32 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.644241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2765bb1f-8b43-4b1b-a9c9-65595f4cea32 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.644528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2765bb1f-8b43-4b1b-a9c9-65595f4cea32 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.645221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2765bb1f-8b43-4b1b-a9c9-65595f4cea32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.647146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2765bb1f-8b43-4b1b-a9c9-65595f4cea32 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.670300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1025/2029] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:48 2022] POST /identity/v3/users => generated 299 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:48.675192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfcacc0-d871-42bb-aae4-141d7cd7b860 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.685930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfcacc0-d871-42bb-aae4-141d7cd7b860 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.691030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecef4950-a94c-4769-8d9c-b7b21bbbd6ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.691714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecef4950-a94c-4769-8d9c-b7b21bbbd6ff admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.692006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecef4950-a94c-4769-8d9c-b7b21bbbd6ff admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.692287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecef4950-a94c-4769-8d9c-b7b21bbbd6ff admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.693053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecef4950-a94c-4769-8d9c-b7b21bbbd6ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.694673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecef4950-a94c-4769-8d9c-b7b21bbbd6ff admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.709522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1005/2030] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:48.710624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2765bb1f-8b43-4b1b-a9c9-65595f4cea32 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.721068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2765bb1f-8b43-4b1b-a9c9-65595f4cea32 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.726056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be907f-6b94-499c-a1ae-acc814b2f6aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.726771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be907f-6b94-499c-a1ae-acc814b2f6aa admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.727072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be907f-6b94-499c-a1ae-acc814b2f6aa admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.727357 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32be907f-6b94-499c-a1ae-acc814b2f6aa admin admin] PATH_INFO: `/v3/projects/e8053cb117964334bbceea3809478636/users/f2a73ffdc4604b159e786ab9e71c52a9/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.743851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32be907f-6b94-499c-a1ae-acc814b2f6aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8053cb117964334bbceea3809478636, user_id=f2a73ffdc4604b159e786ab9e71c52a9, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.746593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32be907f-6b94-499c-a1ae-acc814b2f6aa admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.760273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1026/2031] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:48 2022] PUT /identity/v3/projects/e8053cb117964334bbceea3809478636/users/f2a73ffdc4604b159e786ab9e71c52a9/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:48.764226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecef4950-a94c-4769-8d9c-b7b21bbbd6ff admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.775442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecef4950-a94c-4769-8d9c-b7b21bbbd6ff admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.797546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:48.846288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0749e53f-b4e7-4484-8058-0e010a33a32c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.846288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0749e53f-b4e7-4484-8058-0e010a33a32c admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.846288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0749e53f-b4e7-4484-8058-0e010a33a32c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.846288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0749e53f-b4e7-4484-8058-0e010a33a32c admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.846858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0749e53f-b4e7-4484-8058-0e010a33a32c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.846858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0749e53f-b4e7-4484-8058-0e010a33a32c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.855355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1006/2032] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:48 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:48.859267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be907f-6b94-499c-a1ae-acc814b2f6aa admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:48.865639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:48.873038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32be907f-6b94-499c-a1ae-acc814b2f6aa admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:48.878776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db387a5-d3ca-4a15-9931-4745551d89f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:48.879518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db387a5-d3ca-4a15-9931-4745551d89f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.879875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db387a5-d3ca-4a15-9931-4745551d89f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.880225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db387a5-d3ca-4a15-9931-4745551d89f4 admin admin] PATH_INFO: `/v3/projects/e8053cb117964334bbceea3809478636/users/f2a73ffdc4604b159e786ab9e71c52a9/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.882319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db387a5-d3ca-4a15-9931-4745551d89f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8053cb117964334bbceea3809478636, user_id=f2a73ffdc4604b159e786ab9e71c52a9, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:48.885062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db387a5-d3ca-4a15-9931-4745551d89f4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:48.896328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1027/2033] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:48 2022] PUT /identity/v3/projects/e8053cb117964334bbceea3809478636/users/f2a73ffdc4604b159e786ab9e71c52a9/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:48.964922 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945fa6a4-2c12-4af6-a453-8b848258ccd5 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:48.965126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945fa6a4-2c12-4af6-a453-8b848258ccd5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:48.965394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945fa6a4-2c12-4af6-a453-8b848258ccd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:48.993655 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-945fa6a4-2c12-4af6-a453-8b848258ccd5 None None] MFA Rules not processed for user `f2a73ffdc4604b159e786ab9e71c52a9`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:48.994875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-945fa6a4-2c12-4af6-a453-8b848258ccd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:49.062424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1007/2034] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:48 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:49.073588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db387a5-d3ca-4a15-9931-4745551d89f4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:49.075247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9db387a5-d3ca-4a15-9931-4745551d89f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:49.134266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db387a5-d3ca-4a15-9931-4745551d89f4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:49.137888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2f2d38-58d8-47db-9999-921f3b6101e4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:49.138504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f2d38-58d8-47db-9999-921f3b6101e4 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:49.138785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f2d38-58d8-47db-9999-921f3b6101e4 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:49.139056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f2d38-58d8-47db-9999-921f3b6101e4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:49.147816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2f2d38-58d8-47db-9999-921f3b6101e4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:49.149907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce2f2d38-58d8-47db-9999-921f3b6101e4 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:49.159833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1028/2035] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:49 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:49.782320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945fa6a4-2c12-4af6-a453-8b848258ccd5 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:49.844538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945fa6a4-2c12-4af6-a453-8b848258ccd5 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:49.850619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:49.851277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:49.851566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:49.851849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] PATH_INFO: `/v3/users/d7b042c54b5143519ef94543256fb57b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:49.853059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7b042c54b5143519ef94543256fb57b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:49.854987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:49.908358 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7b042c54b5143519ef94543256fb57b'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:49.910118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:49.918087 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7b042c54b5143519ef94543256fb57b'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:49.919663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7b042c54b5143519ef94543256fb57b'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:49.921040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7b042c54b5143519ef94543256fb57b'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:49.930217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7b042c54b5143519ef94543256fb57b'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:49.956476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1008/2036] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:49 2022] DELETE /identity/v3/users/d7b042c54b5143519ef94543256fb57b => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:49.960983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007ea620-2731-4b1e-aea4-76b5fc27bd1e None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:49.961453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007ea620-2731-4b1e-aea4-76b5fc27bd1e None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:49.961545 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007ea620-2731-4b1e-aea4-76b5fc27bd1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:49.981693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-007ea620-2731-4b1e-aea4-76b5fc27bd1e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:49.982844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-007ea620-2731-4b1e-aea4-76b5fc27bd1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:50.044122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1029/2037] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:50.054556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:50.055964 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:50.112056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c38a8a7-38e4-41ca-8a39-6ea7db003722 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:50.115724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee62efe-23f4-48f1-aabc-1d919dcfe7a2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:50.116344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee62efe-23f4-48f1-aabc-1d919dcfe7a2 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:50.116650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee62efe-23f4-48f1-aabc-1d919dcfe7a2 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:50.116934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee62efe-23f4-48f1-aabc-1d919dcfe7a2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:50.126360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee62efe-23f4-48f1-aabc-1d919dcfe7a2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:50.128661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee62efe-23f4-48f1-aabc-1d919dcfe7a2 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:50.139424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1009/2038] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:50.412526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007ea620-2731-4b1e-aea4-76b5fc27bd1e None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:50.414068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-007ea620-2731-4b1e-aea4-76b5fc27bd1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:50.427806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007ea620-2731-4b1e-aea4-76b5fc27bd1e None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:50.432612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:50.433710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:50.434011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:50.434363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] PATH_INFO: `/v3/projects/4093a675559440daa934166089660e0e` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:50.435530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4093a675559440daa934166089660e0e)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:50.437408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:50.484931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4093a675559440daa934166089660e0e', 'request_id': 'req-7cecb6c0-c619-459b-8538-ea80782c8e06'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:50.492413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4093a675559440daa934166089660e0e', 'request_id': 'req-7cecb6c0-c619-459b-8538-ea80782c8e06'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:50.500035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4093a675559440daa934166089660e0e', 'request_id': 'req-7cecb6c0-c619-459b-8538-ea80782c8e06'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:50.511205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] The token cache is being invalidate because project 4093a675559440daa934166089660e0e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:50.511498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:50.512127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:50.513690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1030/2039] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:50 2022] DELETE /identity/v3/projects/4093a675559440daa934166089660e0e => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:50.517697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee62efe-23f4-48f1-aabc-1d919dcfe7a2 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:50.539802 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee62efe-23f4-48f1-aabc-1d919dcfe7a2 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:50.608511 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:50.609203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:50.609490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:50.609771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] PATH_INFO: `/v3/users/f2a73ffdc4604b159e786ab9e71c52a9` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:50.612138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2a73ffdc4604b159e786ab9e71c52a9)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:50.614163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:50.688260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2a73ffdc4604b159e786ab9e71c52a9'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:50.699071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2a73ffdc4604b159e786ab9e71c52a9'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:50.700274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2a73ffdc4604b159e786ab9e71c52a9'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:50.701280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2a73ffdc4604b159e786ab9e71c52a9'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:50.725209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2a73ffdc4604b159e786ab9e71c52a9'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:50.731831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:INSERT=2,keystone:DELETE=12,keystone:SELECT=86 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:50.751120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1010/2040] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:50 2022] DELETE /identity/v3/users/f2a73ffdc4604b159e786ab9e71c52a9 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:51.055733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:51.057344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:51.075287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cecb6c0-c619-459b-8538-ea80782c8e06 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:51.106360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:51.182532 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:51.183193 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:51.183500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:51.183766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] PATH_INFO: `/v3/projects/e8053cb117964334bbceea3809478636` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:51.184996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8053cb117964334bbceea3809478636)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:51.186949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:51.245902 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8053cb117964334bbceea3809478636', 'request_id': 'req-a9c5f22b-eb12-4e70-aff1-f0227943d302'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:51.254617 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8053cb117964334bbceea3809478636', 'request_id': 'req-a9c5f22b-eb12-4e70-aff1-f0227943d302'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:51.265196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8053cb117964334bbceea3809478636', 'request_id': 'req-a9c5f22b-eb12-4e70-aff1-f0227943d302'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:51.277771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] The token cache is being invalidate because project e8053cb117964334bbceea3809478636 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:51.278095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:51.278873 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a9c5f22b-eb12-4e70-aff1-f0227943d302 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:51.280432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1031/2041] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:51 2022] DELETE /identity/v3/projects/e8053cb117964334bbceea3809478636 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:52.260329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:52.262072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:52.293966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514d3629-71c1-46aa-984b-ea0d84af30b9 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:52.384559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:52.386214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:52.386659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:52.387094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] PATH_INFO: `/v3/users/35bd7418b6f24c30b2b7e2aadf11aa85` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:52.389009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=35bd7418b6f24c30b2b7e2aadf11aa85)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:52.391747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:52.473907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35bd7418b6f24c30b2b7e2aadf11aa85'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:52.483363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35bd7418b6f24c30b2b7e2aadf11aa85'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:52.484158 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35bd7418b6f24c30b2b7e2aadf11aa85'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:52.486958 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35bd7418b6f24c30b2b7e2aadf11aa85'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:52.497642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35bd7418b6f24c30b2b7e2aadf11aa85'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:52.521890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1011/2042] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:52 2022] DELETE /identity/v3/users/35bd7418b6f24c30b2b7e2aadf11aa85 => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:52.526912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86140f5-48d6-4a33-aed5-4d3a0b963525 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:52.527409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86140f5-48d6-4a33-aed5-4d3a0b963525 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:52.527698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86140f5-48d6-4a33-aed5-4d3a0b963525 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:52.550468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d86140f5-48d6-4a33-aed5-4d3a0b963525 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:52.551916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d86140f5-48d6-4a33-aed5-4d3a0b963525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:52.618324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1032/2043] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:52.638797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:52.640298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:52.715152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9140a879-f99d-417a-a113-2d3603a934dc admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:52.719628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31751a3-97cf-4e5b-8c88-87978db3a470 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:52.721388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31751a3-97cf-4e5b-8c88-87978db3a470 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:52.728897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31751a3-97cf-4e5b-8c88-87978db3a470 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:52.731110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31751a3-97cf-4e5b-8c88-87978db3a470 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:52.746428 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31751a3-97cf-4e5b-8c88-87978db3a470 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:52.748224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31751a3-97cf-4e5b-8c88-87978db3a470 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:52.764239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1012/2044] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:48:54.166817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86140f5-48d6-4a33-aed5-4d3a0b963525 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:54.168483 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d86140f5-48d6-4a33-aed5-4d3a0b963525 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:54.193875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86140f5-48d6-4a33-aed5-4d3a0b963525 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:54.205425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:54.206986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:54.209916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:54.210417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] PATH_INFO: `/v3/projects/03046bc3e7ec4b50b15e0479d1da89d2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:54.213147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] RBAC: Authorizing `identity:delete_project(project_id=03046bc3e7ec4b50b15e0479d1da89d2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:54.217493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:54.287200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:54.316826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03046bc3e7ec4b50b15e0479d1da89d2', 'request_id': 'req-e26a0df8-173d-4566-9f03-ba093697955f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:54.343464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03046bc3e7ec4b50b15e0479d1da89d2', 'request_id': 'req-e26a0df8-173d-4566-9f03-ba093697955f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:54.367264 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03046bc3e7ec4b50b15e0479d1da89d2', 'request_id': 'req-e26a0df8-173d-4566-9f03-ba093697955f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:54.385395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] The token cache is being invalidate because project 03046bc3e7ec4b50b15e0479d1da89d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:54.385796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:54.388185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:54.393971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1033/2045] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:54 2022] DELETE /identity/v3/projects/03046bc3e7ec4b50b15e0479d1da89d2 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:54.394405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31751a3-97cf-4e5b-8c88-87978db3a470 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:54.414273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31751a3-97cf-4e5b-8c88-87978db3a470 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:54.420280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:54.506403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:54.507071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:54.507822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:54.508202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] PATH_INFO: `/v3/users/f78412ae0dab499d9bd52431a738aa19` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:54.510404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] RBAC: Authorizing `identity:delete_user(user_id=f78412ae0dab499d9bd52431a738aa19)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:54.512515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:54.573517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f78412ae0dab499d9bd52431a738aa19'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:54.583782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f78412ae0dab499d9bd52431a738aa19'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:54.584719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f78412ae0dab499d9bd52431a738aa19'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:54.585415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f78412ae0dab499d9bd52431a738aa19'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:54.602310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0da40c2f-b3b0-43d0-b2be-386fefaa83de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f78412ae0dab499d9bd52431a738aa19'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:54.632865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1013/2046] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:54 2022] DELETE /identity/v3/users/f78412ae0dab499d9bd52431a738aa19 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:55.066350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:55.067795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:55.094915 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26a0df8-173d-4566-9f03-ba093697955f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:55.207011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:55.207742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:55.208255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:55.208844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] PATH_INFO: `/v3/projects/15dcab582c2c405bbc665085700277d9` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:55.210383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] RBAC: Authorizing `identity:delete_project(project_id=15dcab582c2c405bbc665085700277d9)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:55.213033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:55.277746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15dcab582c2c405bbc665085700277d9', 'request_id': 'req-948dd439-b02c-4d62-94e9-bad69cf3505d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:55.284806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15dcab582c2c405bbc665085700277d9', 'request_id': 'req-948dd439-b02c-4d62-94e9-bad69cf3505d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:55.293733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15dcab582c2c405bbc665085700277d9', 'request_id': 'req-948dd439-b02c-4d62-94e9-bad69cf3505d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:55.307256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] The token cache is being invalidate because project 15dcab582c2c405bbc665085700277d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:55.307548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:55.308205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-948dd439-b02c-4d62-94e9-bad69cf3505d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:55.310137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1034/2047] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:55 2022] DELETE /identity/v3/projects/15dcab582c2c405bbc665085700277d9 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:55.316005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db73d06-c32f-43e8-a158-4a8dd20481a5 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:55.316817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db73d06-c32f-43e8-a158-4a8dd20481a5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:55.317153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db73d06-c32f-43e8-a158-4a8dd20481a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:55.355610 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-1db73d06-c32f-43e8-a158-4a8dd20481a5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:55.356882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-1db73d06-c32f-43e8-a158-4a8dd20481a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:55.452395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a0fefe-6107-4f6b-afed-7b205b07aa7a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:55.452712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a0fefe-6107-4f6b-afed-7b205b07aa7a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:55.452992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a0fefe-6107-4f6b-afed-7b205b07aa7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:55.465570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1014/2048] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:55.481540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-38a0fefe-6107-4f6b-afed-7b205b07aa7a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:55.482865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-38a0fefe-6107-4f6b-afed-7b205b07aa7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:55.497452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db73d06-c32f-43e8-a158-4a8dd20481a5 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:55.504790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1035/2049] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:55.519929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db73d06-c32f-43e8-a158-4a8dd20481a5 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:55.536825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426dcea9-301e-42c0-b70b-838ada43a643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:55.537491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426dcea9-301e-42c0-b70b-838ada43a643 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:55.537889 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426dcea9-301e-42c0-b70b-838ada43a643 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:55.538237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426dcea9-301e-42c0-b70b-838ada43a643 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:55.539041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426dcea9-301e-42c0-b70b-838ada43a643 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:55.539415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426dcea9-301e-42c0-b70b-838ada43a643 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:55.541353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426dcea9-301e-42c0-b70b-838ada43a643 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:55.574624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a0fefe-6107-4f6b-afed-7b205b07aa7a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:55.582146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1015/2050] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:48:55.586735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a0fefe-6107-4f6b-afed-7b205b07aa7a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:55.589827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426dcea9-301e-42c0-b70b-838ada43a643 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:55.591794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-426dcea9-301e-42c0-b70b-838ada43a643 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:55.599814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee7c53e-b09c-40d8-a863-8070e6944213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:55.601444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee7c53e-b09c-40d8-a863-8070e6944213 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:55.601741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee7c53e-b09c-40d8-a863-8070e6944213 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:55.602021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee7c53e-b09c-40d8-a863-8070e6944213 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:55.602703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee7c53e-b09c-40d8-a863-8070e6944213 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:55.609345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee7c53e-b09c-40d8-a863-8070e6944213 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:55.624120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426dcea9-301e-42c0-b70b-838ada43a643 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:55.636811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:55.638369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:55.638664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:55.639587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] PATH_INFO: `/v3/users/5591e354be1e461fb5dcbe8a7ddb2cdd` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:55.641887 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] RBAC: Authorizing `identity:delete_user(user_id=5591e354be1e461fb5dcbe8a7ddb2cdd)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:55.644281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:55.652232 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1036/2051] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:55 2022] POST /identity/v3/projects => generated 369 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:55.654395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee7c53e-b09c-40d8-a863-8070e6944213 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:55.675837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:55.722081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:55.735090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5591e354be1e461fb5dcbe8a7ddb2cdd'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:55.748493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5591e354be1e461fb5dcbe8a7ddb2cdd'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:55.749625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5591e354be1e461fb5dcbe8a7ddb2cdd'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:55.750926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5591e354be1e461fb5dcbe8a7ddb2cdd'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:55.764177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-77bd7684-7617-48cf-be55-edd9d834a86a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5591e354be1e461fb5dcbe8a7ddb2cdd'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:55.778675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee7c53e-b09c-40d8-a863-8070e6944213 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:55.785645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57715446-f480-45e6-9e74-cc7a638b8935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:55.786550 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57715446-f480-45e6-9e74-cc7a638b8935 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:55.787284 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57715446-f480-45e6-9e74-cc7a638b8935 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:55.787567 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57715446-f480-45e6-9e74-cc7a638b8935 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:55.788362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57715446-f480-45e6-9e74-cc7a638b8935 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:55.791377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57715446-f480-45e6-9e74-cc7a638b8935 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:55.794011 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1016/2052] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:48:55 2022] DELETE /identity/v3/users/5591e354be1e461fb5dcbe8a7ddb2cdd => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:55.800249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737f329e-b70f-4bd7-b8a9-85dfe9dcf9ea None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:55.800552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737f329e-b70f-4bd7-b8a9-85dfe9dcf9ea None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:55.803967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737f329e-b70f-4bd7-b8a9-85dfe9dcf9ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:55.824265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1037/2053] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:55 2022] POST /identity/v3/users => generated 285 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:55.829083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57715446-f480-45e6-9e74-cc7a638b8935 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:55.832009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-57715446-f480-45e6-9e74-cc7a638b8935 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:55.835956 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-737f329e-b70f-4bd7-b8a9-85dfe9dcf9ea None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:55.838017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-737f329e-b70f-4bd7-b8a9-85dfe9dcf9ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:55.851034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57715446-f480-45e6-9e74-cc7a638b8935 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:55.937941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9f9a14-35a7-461f-a04f-0eb1a387a2ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:55.940030 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9f9a14-35a7-461f-a04f-0eb1a387a2ae admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:55.940320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9f9a14-35a7-461f-a04f-0eb1a387a2ae admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:55.940641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9f9a14-35a7-461f-a04f-0eb1a387a2ae admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:55.941392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9f9a14-35a7-461f-a04f-0eb1a387a2ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:55.943206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9f9a14-35a7-461f-a04f-0eb1a387a2ae admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:55.959222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1038/2054] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:55 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:55.962668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9f9a14-35a7-461f-a04f-0eb1a387a2ae admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:55.966670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1017/2055] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:55.976411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737f329e-b70f-4bd7-b8a9-85dfe9dcf9ea None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:55.978247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-737f329e-b70f-4bd7-b8a9-85dfe9dcf9ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:55.986360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9f9a14-35a7-461f-a04f-0eb1a387a2ae admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:55.992159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a84537-be15-44d5-92dc-5a8e961a3e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:55.992432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a84537-be15-44d5-92dc-5a8e961a3e71 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:55.992753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a84537-be15-44d5-92dc-5a8e961a3e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:55.993029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a84537-be15-44d5-92dc-5a8e961a3e71 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:55.993993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a84537-be15-44d5-92dc-5a8e961a3e71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:55.995389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a84537-be15-44d5-92dc-5a8e961a3e71 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.012131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1039/2056] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:55 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:56.014058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a84537-be15-44d5-92dc-5a8e961a3e71 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:56.025407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a84537-be15-44d5-92dc-5a8e961a3e71 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.031137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9311bdda-3636-49de-a898-9e6ec60a1bf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.031777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9311bdda-3636-49de-a898-9e6ec60a1bf9 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.032064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9311bdda-3636-49de-a898-9e6ec60a1bf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.032345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9311bdda-3636-49de-a898-9e6ec60a1bf9 admin admin] PATH_INFO: `/v3/projects/614cff68b07c457fb1da6fddccf29f41/users/28a18cb55d7044cf90ca8cf1e158a363/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.041907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737f329e-b70f-4bd7-b8a9-85dfe9dcf9ea None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.046010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636a0585-b1ca-47c5-8641-fed9534695c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.046737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636a0585-b1ca-47c5-8641-fed9534695c7 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.047096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636a0585-b1ca-47c5-8641-fed9534695c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.047454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636a0585-b1ca-47c5-8641-fed9534695c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.048665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9311bdda-3636-49de-a898-9e6ec60a1bf9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=614cff68b07c457fb1da6fddccf29f41, user_id=28a18cb55d7044cf90ca8cf1e158a363, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.051471 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9311bdda-3636-49de-a898-9e6ec60a1bf9 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.059838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636a0585-b1ca-47c5-8641-fed9534695c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.061333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636a0585-b1ca-47c5-8641-fed9534695c7 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.070173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1040/2057] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:56 2022] PUT /identity/v3/projects/614cff68b07c457fb1da6fddccf29f41/users/28a18cb55d7044cf90ca8cf1e158a363/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:48:56.072542 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1018/2058] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:56.082190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bdd2ba-58ab-42e1-ad8b-1f955338d4f8 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.082493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bdd2ba-58ab-42e1-ad8b-1f955338d4f8 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.082779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bdd2ba-58ab-42e1-ad8b-1f955338d4f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.115971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-00bdd2ba-58ab-42e1-ad8b-1f955338d4f8 None None] MFA Rules not processed for user `28a18cb55d7044cf90ca8cf1e158a363`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:56.117288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-00bdd2ba-58ab-42e1-ad8b-1f955338d4f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:56.200507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1041/2059] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:56 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:56.205535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636a0585-b1ca-47c5-8641-fed9534695c7 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:56.218733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636a0585-b1ca-47c5-8641-fed9534695c7 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.260387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:56.298695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abafcd9-db5c-4015-98fa-6e6d8f691062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.300403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abafcd9-db5c-4015-98fa-6e6d8f691062 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.300843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abafcd9-db5c-4015-98fa-6e6d8f691062 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.301564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abafcd9-db5c-4015-98fa-6e6d8f691062 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.302006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abafcd9-db5c-4015-98fa-6e6d8f691062 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.304221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abafcd9-db5c-4015-98fa-6e6d8f691062 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.344183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1019/2060] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:56 2022] POST /identity/v3/projects => generated 369 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:56.350490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bdd2ba-58ab-42e1-ad8b-1f955338d4f8 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:56.366821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bdd2ba-58ab-42e1-ad8b-1f955338d4f8 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.378376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:56.390305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abafcd9-db5c-4015-98fa-6e6d8f691062 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:56.392788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-1abafcd9-db5c-4015-98fa-6e6d8f691062 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:56.475248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd0c505-3c30-4345-8ccc-f835f998e1f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.476302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0c505-3c30-4345-8ccc-f835f998e1f9 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.476712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0c505-3c30-4345-8ccc-f835f998e1f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.477088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0c505-3c30-4345-8ccc-f835f998e1f9 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.477858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd0c505-3c30-4345-8ccc-f835f998e1f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.479920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd0c505-3c30-4345-8ccc-f835f998e1f9 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.485103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abafcd9-db5c-4015-98fa-6e6d8f691062 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.490683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.491574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.491926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.492276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] PATH_INFO: `/v3/projects/be01946c207e4739a5d95502633ce87d` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.493548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=be01946c207e4739a5d95502633ce87d)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.495613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.513371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1042/2061] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:56 2022] POST /identity/v3/users => generated 291 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:56.517521 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd0c505-3c30-4345-8ccc-f835f998e1f9 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:56.528016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd0c505-3c30-4345-8ccc-f835f998e1f9 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.534632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f916442-7ecf-4902-8a83-accc5afa15d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.535340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f916442-7ecf-4902-8a83-accc5afa15d3 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.536348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f916442-7ecf-4902-8a83-accc5afa15d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.536595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f916442-7ecf-4902-8a83-accc5afa15d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.537969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f916442-7ecf-4902-8a83-accc5afa15d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.541026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f916442-7ecf-4902-8a83-accc5afa15d3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.558995 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1043/2062] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:56 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:56.563313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f916442-7ecf-4902-8a83-accc5afa15d3 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:56.567081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be01946c207e4739a5d95502633ce87d', 'request_id': 'req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:56.575062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f916442-7ecf-4902-8a83-accc5afa15d3 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.582050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be01946c207e4739a5d95502633ce87d', 'request_id': 'req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:56.593692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be01946c207e4739a5d95502633ce87d', 'request_id': 'req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:56.616243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] The token cache is being invalidate because project be01946c207e4739a5d95502633ce87d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:48:56.617969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:56.618286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1d6caab7-45ba-49fb-9b8d-ad8b8c5f51d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:48:56.622184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1020/2063] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:48:56 2022] DELETE /identity/v3/projects/be01946c207e4739a5d95502633ce87d => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:56.636990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdde7f50-93a7-440f-b2ed-900f51509507 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.636990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdde7f50-93a7-440f-b2ed-900f51509507 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.637289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdde7f50-93a7-440f-b2ed-900f51509507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.673790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fe39ee-0340-477e-ba51-4f4054d632ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.674682 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe39ee-0340-477e-ba51-4f4054d632ac admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.675016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe39ee-0340-477e-ba51-4f4054d632ac admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.675445 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fe39ee-0340-477e-ba51-4f4054d632ac admin admin] PATH_INFO: `/v3/projects/4a6c4b304e7a4b3a89e265d0bb9ffa46/users/19a1a8c4a4c943b1bbddc3316f79b14b/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.695111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fe39ee-0340-477e-ba51-4f4054d632ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a6c4b304e7a4b3a89e265d0bb9ffa46, user_id=19a1a8c4a4c943b1bbddc3316f79b14b, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.698796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fe39ee-0340-477e-ba51-4f4054d632ac admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.704628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-fdde7f50-93a7-440f-b2ed-900f51509507 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:56.708220 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fdde7f50-93a7-440f-b2ed-900f51509507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:56.728662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1044/2064] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:56 2022] PUT /identity/v3/projects/4a6c4b304e7a4b3a89e265d0bb9ffa46/users/19a1a8c4a4c943b1bbddc3316f79b14b/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:56.732952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fe39ee-0340-477e-ba51-4f4054d632ac admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:56.737320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-04fe39ee-0340-477e-ba51-4f4054d632ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:56.800668 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:56.835432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1021/2065] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:56.836878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fdae49-7044-4b66-953d-7929e01ab3e9 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.837175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fdae49-7044-4b66-953d-7929e01ab3e9 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.837458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fdae49-7044-4b66-953d-7929e01ab3e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.865864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-79fdae49-7044-4b66-953d-7929e01ab3e9 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:56.867172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-79fdae49-7044-4b66-953d-7929e01ab3e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:56.875753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1022/2066] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:56.878721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fe39ee-0340-477e-ba51-4f4054d632ac admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.880663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fdae49-7044-4b66-953d-7929e01ab3e9 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:56.884667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6561c70b-9fbb-4a6d-b422-9fb9f43b2733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.885379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6561c70b-9fbb-4a6d-b422-9fb9f43b2733 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.885742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6561c70b-9fbb-4a6d-b422-9fb9f43b2733 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.886110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6561c70b-9fbb-4a6d-b422-9fb9f43b2733 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.886891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6561c70b-9fbb-4a6d-b422-9fb9f43b2733 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.888643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6561c70b-9fbb-4a6d-b422-9fb9f43b2733 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.891975 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fdae49-7044-4b66-953d-7929e01ab3e9 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.902530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1045/2067] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:56 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:56.902716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b1a9be-1ca9-49a6-a7b5-c01d65ce8109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.902716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b1a9be-1ca9-49a6-a7b5-c01d65ce8109 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.902716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b1a9be-1ca9-49a6-a7b5-c01d65ce8109 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.902716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b1a9be-1ca9-49a6-a7b5-c01d65ce8109 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.903013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b1a9be-1ca9-49a6-a7b5-c01d65ce8109 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:48:56.903461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b1a9be-1ca9-49a6-a7b5-c01d65ce8109 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.904864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b1a9be-1ca9-49a6-a7b5-c01d65ce8109 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.907852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6561c70b-9fbb-4a6d-b422-9fb9f43b2733 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:56.920768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6561c70b-9fbb-4a6d-b422-9fb9f43b2733 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.930824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8cc01d5-8cf0-44a4-b2d5-541c1c2c88df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.931021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1023/2068] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:48:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:48:56.931119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc01d5-8cf0-44a4-b2d5-541c1c2c88df admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.931119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc01d5-8cf0-44a4-b2d5-541c1c2c88df admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.931119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8cc01d5-8cf0-44a4-b2d5-541c1c2c88df admin admin] PATH_INFO: `/v3/projects/4a6c4b304e7a4b3a89e265d0bb9ffa46/users/19a1a8c4a4c943b1bbddc3316f79b14b/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.932496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b1a9be-1ca9-49a6-a7b5-c01d65ce8109 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:56.933471 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cc01d5-8cf0-44a4-b2d5-541c1c2c88df admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a6c4b304e7a4b3a89e265d0bb9ffa46, user_id=19a1a8c4a4c943b1bbddc3316f79b14b, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.937794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8cc01d5-8cf0-44a4-b2d5-541c1c2c88df admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.942814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:56.944179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b1a9be-1ca9-49a6-a7b5-c01d65ce8109 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:56.949892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c36356-45d2-4996-bbd8-145a795b546e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:56.950533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c36356-45d2-4996-bbd8-145a795b546e admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.950961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c36356-45d2-4996-bbd8-145a795b546e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.951247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c36356-45d2-4996-bbd8-145a795b546e admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.951928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c36356-45d2-4996-bbd8-145a795b546e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:56.953875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c36356-45d2-4996-bbd8-145a795b546e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:56.957827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1046/2069] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:56 2022] PUT /identity/v3/projects/4a6c4b304e7a4b3a89e265d0bb9ffa46/users/19a1a8c4a4c943b1bbddc3316f79b14b/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:56.968233 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c533c3-ba5f-4003-af3f-dd3e1d162481 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:56.968529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c533c3-ba5f-4003-af3f-dd3e1d162481 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:56.968851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c533c3-ba5f-4003-af3f-dd3e1d162481 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:56.988685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1024/2070] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:56 2022] POST /identity/v3/projects => generated 375 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:56.990962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c36356-45d2-4996-bbd8-145a795b546e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.005434 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c2c533c3-ba5f-4003-af3f-dd3e1d162481 None None] MFA Rules not processed for user `19a1a8c4a4c943b1bbddc3316f79b14b`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:57.006713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c2c533c3-ba5f-4003-af3f-dd3e1d162481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:57.072401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c36356-45d2-4996-bbd8-145a795b546e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.078336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21b1459-56d3-4cbb-a63f-0f81473bb8f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.079951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21b1459-56d3-4cbb-a63f-0f81473bb8f2 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.080409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21b1459-56d3-4cbb-a63f-0f81473bb8f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.080867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21b1459-56d3-4cbb-a63f-0f81473bb8f2 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.081816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21b1459-56d3-4cbb-a63f-0f81473bb8f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.083464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1047/2071] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:56 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:57.084476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21b1459-56d3-4cbb-a63f-0f81473bb8f2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:57.096519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c533c3-ba5f-4003-af3f-dd3e1d162481 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.099597 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c2c533c3-ba5f-4003-af3f-dd3e1d162481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:57.110852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1025/2072] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:56 2022] POST /identity/v3/users => generated 288 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:57.114928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21b1459-56d3-4cbb-a63f-0f81473bb8f2 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.125786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21b1459-56d3-4cbb-a63f-0f81473bb8f2 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.131130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa6bd50-27d6-4479-8837-220de5a5b5a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.131833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa6bd50-27d6-4479-8837-220de5a5b5a9 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.132315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa6bd50-27d6-4479-8837-220de5a5b5a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.132658 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa6bd50-27d6-4479-8837-220de5a5b5a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.133337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa6bd50-27d6-4479-8837-220de5a5b5a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.134953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffa6bd50-27d6-4479-8837-220de5a5b5a9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:57.160548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1026/2073] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:57 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:48:57.161918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa6bd50-27d6-4479-8837-220de5a5b5a9 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.169236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c533c3-ba5f-4003-af3f-dd3e1d162481 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.175425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e99551-0b90-4b11-b68c-c4a7d9a8257d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.175961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa6bd50-27d6-4479-8837-220de5a5b5a9 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.176522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e99551-0b90-4b11-b68c-c4a7d9a8257d service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.177261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e99551-0b90-4b11-b68c-c4a7d9a8257d service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.177738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e99551-0b90-4b11-b68c-c4a7d9a8257d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.181966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6c621-f74c-4d4c-97ed-b834f8d8f523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.182638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6c621-f74c-4d4c-97ed-b834f8d8f523 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.182975 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6c621-f74c-4d4c-97ed-b834f8d8f523 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.183267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b6c621-f74c-4d4c-97ed-b834f8d8f523 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.184146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b6c621-f74c-4d4c-97ed-b834f8d8f523 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.187846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b6c621-f74c-4d4c-97ed-b834f8d8f523 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:57.191629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e99551-0b90-4b11-b68c-c4a7d9a8257d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.193242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e99551-0b90-4b11-b68c-c4a7d9a8257d service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:57.206138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1027/2074] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:57 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:57.208525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1048/2075] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:57 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:57.209143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6c621-f74c-4d4c-97ed-b834f8d8f523 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.223623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b6c621-f74c-4d4c-97ed-b834f8d8f523 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.234010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d061f7d-ff65-4247-b54d-fe9575948cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.235331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d061f7d-ff65-4247-b54d-fe9575948cd3 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.236283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d061f7d-ff65-4247-b54d-fe9575948cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.237037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d061f7d-ff65-4247-b54d-fe9575948cd3 admin admin] PATH_INFO: `/v3/projects/301f15b6723146d99b477817ffd324a1/users/956854812e1e4010aeeb064237bdcf0c/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.258273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d061f7d-ff65-4247-b54d-fe9575948cd3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=301f15b6723146d99b477817ffd324a1, user_id=956854812e1e4010aeeb064237bdcf0c, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.267992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d061f7d-ff65-4247-b54d-fe9575948cd3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:57.279101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e99551-0b90-4b11-b68c-c4a7d9a8257d service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.281437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-82e99551-0b90-4b11-b68c-c4a7d9a8257d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:57.286022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1028/2076] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:57 2022] PUT /identity/v3/projects/301f15b6723146d99b477817ffd324a1/users/956854812e1e4010aeeb064237bdcf0c/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:57.296235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e99551-0b90-4b11-b68c-c4a7d9a8257d service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.297515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f14fb11-7308-47cd-a4a8-95cf6439e456 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.298640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f14fb11-7308-47cd-a4a8-95cf6439e456 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.299230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f14fb11-7308-47cd-a4a8-95cf6439e456 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.335444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4f14fb11-7308-47cd-a4a8-95cf6439e456 None None] MFA Rules not processed for user `956854812e1e4010aeeb064237bdcf0c`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:57.337000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4f14fb11-7308-47cd-a4a8-95cf6439e456 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:57.377328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f7bff0-25f4-4926-9ad2-9e86fce82e1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.378079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f7bff0-25f4-4926-9ad2-9e86fce82e1f service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.383394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f7bff0-25f4-4926-9ad2-9e86fce82e1f service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.383394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f7bff0-25f4-4926-9ad2-9e86fce82e1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.394940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:57.395919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-43f7bff0-25f4-4926-9ad2-9e86fce82e1f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:57.444214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1029/2077] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:57 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:57.445899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f14fb11-7308-47cd-a4a8-95cf6439e456 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.464869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:57.504735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f7bff0-25f4-4926-9ad2-9e86fce82e1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.506534 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f7bff0-25f4-4926-9ad2-9e86fce82e1f service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:57.533057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1049/2078] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:57 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:48:57.544977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f14fb11-7308-47cd-a4a8-95cf6439e456 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.558181 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db5ee87-f617-4cfe-bb8d-c05d1cbc30ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.558902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db5ee87-f617-4cfe-bb8d-c05d1cbc30ab admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.559197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db5ee87-f617-4cfe-bb8d-c05d1cbc30ab admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.559608 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db5ee87-f617-4cfe-bb8d-c05d1cbc30ab admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.560501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db5ee87-f617-4cfe-bb8d-c05d1cbc30ab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.562713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db5ee87-f617-4cfe-bb8d-c05d1cbc30ab admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:57.619241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1030/2079] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:57 2022] POST /identity/v3/projects => generated 377 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:57.620708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f7bff0-25f4-4926-9ad2-9e86fce82e1f service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.636052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f7bff0-25f4-4926-9ad2-9e86fce82e1f service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.723364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e1cab9-3778-4c97-92e4-5be34ffac8f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.724183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e1cab9-3778-4c97-92e4-5be34ffac8f5 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.724654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e1cab9-3778-4c97-92e4-5be34ffac8f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.725395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e1cab9-3778-4c97-92e4-5be34ffac8f5 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.726279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e1cab9-3778-4c97-92e4-5be34ffac8f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.730177 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e1cab9-3778-4c97-92e4-5be34ffac8f5 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:57.758795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1050/2080] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:57 2022] POST /identity/v3/users => generated 289 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:48:57.761682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db5ee87-f617-4cfe-bb8d-c05d1cbc30ab admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.777182 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db5ee87-f617-4cfe-bb8d-c05d1cbc30ab admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.786335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012b92f6-ed99-448f-98e8-bcb98b8383fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.787184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012b92f6-ed99-448f-98e8-bcb98b8383fb admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.787594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012b92f6-ed99-448f-98e8-bcb98b8383fb admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.788286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012b92f6-ed99-448f-98e8-bcb98b8383fb admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.789408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012b92f6-ed99-448f-98e8-bcb98b8383fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.791779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012b92f6-ed99-448f-98e8-bcb98b8383fb admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:57.816539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1031/2081] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:57 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:57.817893 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e1cab9-3778-4c97-92e4-5be34ffac8f5 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.832825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e1cab9-3778-4c97-92e4-5be34ffac8f5 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.851234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b099d0-5ebe-4735-a212-e30385de072e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.851891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b099d0-5ebe-4735-a212-e30385de072e admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.852877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b099d0-5ebe-4735-a212-e30385de072e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.852877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b099d0-5ebe-4735-a212-e30385de072e admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.853218 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b099d0-5ebe-4735-a212-e30385de072e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.854989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b099d0-5ebe-4735-a212-e30385de072e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:57.880875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1051/2082] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:57 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:57.884682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012b92f6-ed99-448f-98e8-bcb98b8383fb admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:57.901495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012b92f6-ed99-448f-98e8-bcb98b8383fb admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:57.918545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8304bd3f-f704-4587-ad68-2fc7ae1599c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:57.919359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8304bd3f-f704-4587-ad68-2fc7ae1599c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:57.919749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8304bd3f-f704-4587-ad68-2fc7ae1599c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:57.920136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8304bd3f-f704-4587-ad68-2fc7ae1599c1 admin admin] PATH_INFO: `/v3/projects/4d1e691ae5274c78be10dbebdb9d26a0/users/9cc73e663e2a41328342ff95a82c781d/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:57.959043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8304bd3f-f704-4587-ad68-2fc7ae1599c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d1e691ae5274c78be10dbebdb9d26a0, user_id=9cc73e663e2a41328342ff95a82c781d, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:57.965917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8304bd3f-f704-4587-ad68-2fc7ae1599c1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:58.000730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1032/2083] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:57 2022] PUT /identity/v3/projects/4d1e691ae5274c78be10dbebdb9d26a0/users/9cc73e663e2a41328342ff95a82c781d/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:58.011478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea007b74-b269-43e4-a3a7-1311e777c780 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:58.011787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea007b74-b269-43e4-a3a7-1311e777c780 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:58.012073 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea007b74-b269-43e4-a3a7-1311e777c780 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:58.041891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-ea007b74-b269-43e4-a3a7-1311e777c780 None None] MFA Rules not processed for user `9cc73e663e2a41328342ff95a82c781d`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:58.043151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ea007b74-b269-43e4-a3a7-1311e777c780 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:58.128381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1052/2084] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:58 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:48:58.131812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8304bd3f-f704-4587-ad68-2fc7ae1599c1 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:58.214691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8304bd3f-f704-4587-ad68-2fc7ae1599c1 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:58.219585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73f9ef5-5479-4953-9105-b681a7199e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:58.220245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73f9ef5-5479-4953-9105-b681a7199e71 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:58.220708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73f9ef5-5479-4953-9105-b681a7199e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:58.221367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73f9ef5-5479-4953-9105-b681a7199e71 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:58.221722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73f9ef5-5479-4953-9105-b681a7199e71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:58.223785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73f9ef5-5479-4953-9105-b681a7199e71 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:58.260688 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1033/2085] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:48:58 2022] POST /identity/v3/projects => generated 375 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:58.262546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea007b74-b269-43e4-a3a7-1311e777c780 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:58.280572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea007b74-b269-43e4-a3a7-1311e777c780 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:58.328318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:58.343745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23642a81-3365-4e50-be03-b12cb66ee47d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:58.344483 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23642a81-3365-4e50-be03-b12cb66ee47d admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:58.345039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23642a81-3365-4e50-be03-b12cb66ee47d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:58.345390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23642a81-3365-4e50-be03-b12cb66ee47d admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:58.346135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23642a81-3365-4e50-be03-b12cb66ee47d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:58.348092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23642a81-3365-4e50-be03-b12cb66ee47d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:58.370200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1053/2086] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:48:58 2022] POST /identity/v3/users => generated 294 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:48:58.374134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73f9ef5-5479-4953-9105-b681a7199e71 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:58.385587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73f9ef5-5479-4953-9105-b681a7199e71 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:58.390718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ef95a3-9cbe-4434-8034-5b98e62cc198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:58.391303 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ef95a3-9cbe-4434-8034-5b98e62cc198 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:58.391583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ef95a3-9cbe-4434-8034-5b98e62cc198 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:58.392494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ef95a3-9cbe-4434-8034-5b98e62cc198 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:58.393367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ef95a3-9cbe-4434-8034-5b98e62cc198 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:58.394973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ef95a3-9cbe-4434-8034-5b98e62cc198 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:58.406296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1034/2087] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:58.410485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23642a81-3365-4e50-be03-b12cb66ee47d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:58.420965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23642a81-3365-4e50-be03-b12cb66ee47d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:58.425832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1ec1de-c977-4b7c-97f5-9c33b055c430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:58.426503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1ec1de-c977-4b7c-97f5-9c33b055c430 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:58.426809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1ec1de-c977-4b7c-97f5-9c33b055c430 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:58.427081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1ec1de-c977-4b7c-97f5-9c33b055c430 admin admin] PATH_INFO: `/v3/projects/d21b087bf09443bf8d13154cfd339983/users/ca0933cf62df4f30a2d127a834fee2fc/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:58.443887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1ec1de-c977-4b7c-97f5-9c33b055c430 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d21b087bf09443bf8d13154cfd339983, user_id=ca0933cf62df4f30a2d127a834fee2fc, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:58.446634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1ec1de-c977-4b7c-97f5-9c33b055c430 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:58.461390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1054/2088] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:58 2022] PUT /identity/v3/projects/d21b087bf09443bf8d13154cfd339983/users/ca0933cf62df4f30a2d127a834fee2fc/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:58.464711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ef95a3-9cbe-4434-8034-5b98e62cc198 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:58.470097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:48:58.476415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ef95a3-9cbe-4434-8034-5b98e62cc198 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:58.536490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef51f003-f63a-496f-8f9c-7a4c740065f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:58.537157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef51f003-f63a-496f-8f9c-7a4c740065f1 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:58.537449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef51f003-f63a-496f-8f9c-7a4c740065f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:58.537733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef51f003-f63a-496f-8f9c-7a4c740065f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:58.538443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef51f003-f63a-496f-8f9c-7a4c740065f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:58.540349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef51f003-f63a-496f-8f9c-7a4c740065f1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:58.553270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1035/2089] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:48:58 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:48:58.555911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1ec1de-c977-4b7c-97f5-9c33b055c430 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:58.568502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1ec1de-c977-4b7c-97f5-9c33b055c430 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:58.574217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37c29e3-7fe1-43ce-a990-2e9a8a510012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:58.575428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c29e3-7fe1-43ce-a990-2e9a8a510012 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:58.575722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c29e3-7fe1-43ce-a990-2e9a8a510012 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:58.576288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c29e3-7fe1-43ce-a990-2e9a8a510012 admin admin] PATH_INFO: `/v3/projects/d21b087bf09443bf8d13154cfd339983/users/ca0933cf62df4f30a2d127a834fee2fc/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:58.578309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37c29e3-7fe1-43ce-a990-2e9a8a510012 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d21b087bf09443bf8d13154cfd339983, user_id=ca0933cf62df4f30a2d127a834fee2fc, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:58.582214 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37c29e3-7fe1-43ce-a990-2e9a8a510012 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:58.613098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1055/2090] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:48:58 2022] PUT /identity/v3/projects/d21b087bf09443bf8d13154cfd339983/users/ca0933cf62df4f30a2d127a834fee2fc/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:48:58.624964 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70c92ea-ed23-4208-b454-a326493c3c5d None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:58.625331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70c92ea-ed23-4208-b454-a326493c3c5d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:58.625681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70c92ea-ed23-4208-b454-a326493c3c5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:58.655955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-a70c92ea-ed23-4208-b454-a326493c3c5d None None] MFA Rules not processed for user `ca0933cf62df4f30a2d127a834fee2fc`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:48:58.657368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a70c92ea-ed23-4208-b454-a326493c3c5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:48:58.744234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1036/2091] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:48:58 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:48:58.756450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37c29e3-7fe1-43ce-a990-2e9a8a510012 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:48:58.811891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37c29e3-7fe1-43ce-a990-2e9a8a510012 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:48:58.816755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd6f46-f550-457b-8085-78f7fe95fb17 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:48:58.817380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd6f46-f550-457b-8085-78f7fe95fb17 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:48:58.817667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd6f46-f550-457b-8085-78f7fe95fb17 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:48:58.817950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccd6f46-f550-457b-8085-78f7fe95fb17 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:48:58.875415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd6f46-f550-457b-8085-78f7fe95fb17 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:48:58.876946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccd6f46-f550-457b-8085-78f7fe95fb17 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:48:58.886213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1056/2092] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:48:58 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:00.440690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70c92ea-ed23-4208-b454-a326493c3c5d None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:00.450252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70c92ea-ed23-4208-b454-a326493c3c5d None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:00.453829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a06ee19-176a-49ac-94f5-645ff7ad2f28 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:00.455313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a06ee19-176a-49ac-94f5-645ff7ad2f28 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:00.455613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a06ee19-176a-49ac-94f5-645ff7ad2f28 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:00.455892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a06ee19-176a-49ac-94f5-645ff7ad2f28 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:00.520012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a06ee19-176a-49ac-94f5-645ff7ad2f28 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:00.521586 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a06ee19-176a-49ac-94f5-645ff7ad2f28 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:00.536818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1037/2093] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:00 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:01.124396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd6f46-f550-457b-8085-78f7fe95fb17 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:01.146988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a06ee19-176a-49ac-94f5-645ff7ad2f28 service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:01.155672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a06ee19-176a-49ac-94f5-645ff7ad2f28 service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:01.160120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a66f4dbf-aae7-4bc1-85ce-bbe6a6ff68bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:01.160872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f4dbf-aae7-4bc1-85ce-bbe6a6ff68bf service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:01.161162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f4dbf-aae7-4bc1-85ce-bbe6a6ff68bf service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:01.161445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a66f4dbf-aae7-4bc1-85ce-bbe6a6ff68bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:01.171075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66f4dbf-aae7-4bc1-85ce-bbe6a6ff68bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:01.172513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a66f4dbf-aae7-4bc1-85ce-bbe6a6ff68bf service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:01.180167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:01.183221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1038/2094] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:01 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:49:01.194472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccd6f46-f550-457b-8085-78f7fe95fb17 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:01.200344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:01.201137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:01.201424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:01.201767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] PATH_INFO: `/v3/users/28a18cb55d7044cf90ca8cf1e158a363` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:01.203028 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28a18cb55d7044cf90ca8cf1e158a363)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:01.205401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:01.263710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28a18cb55d7044cf90ca8cf1e158a363'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:01.264672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28a18cb55d7044cf90ca8cf1e158a363'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:01.265249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28a18cb55d7044cf90ca8cf1e158a363'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:01.286578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28a18cb55d7044cf90ca8cf1e158a363'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:01.296195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28a18cb55d7044cf90ca8cf1e158a363'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:01.318188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1057/2095] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:01 2022] DELETE /identity/v3/users/28a18cb55d7044cf90ca8cf1e158a363 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:01.322771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d24a88e-d0cd-4989-92b6-263deac0e12f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:01.323069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d24a88e-d0cd-4989-92b6-263deac0e12f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:01.323354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d24a88e-d0cd-4989-92b6-263deac0e12f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:01.343737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-3d24a88e-d0cd-4989-92b6-263deac0e12f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:01.345820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3d24a88e-d0cd-4989-92b6-263deac0e12f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:01.349941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:01.406703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1039/2096] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:01.417029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:01.418466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:01.477411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2962954-41e9-48c2-baeb-3a9f0e562ec4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:01.481066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc62f1d-b4c8-4dc2-ad36-f173d1822a24 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:01.481681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc62f1d-b4c8-4dc2-ad36-f173d1822a24 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:01.481966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc62f1d-b4c8-4dc2-ad36-f173d1822a24 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:01.482245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc62f1d-b4c8-4dc2-ad36-f173d1822a24 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:01.492373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc62f1d-b4c8-4dc2-ad36-f173d1822a24 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:01.493747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc62f1d-b4c8-4dc2-ad36-f173d1822a24 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:01.504417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1058/2097] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:01.802083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d24a88e-d0cd-4989-92b6-263deac0e12f None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:01.803938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3d24a88e-d0cd-4989-92b6-263deac0e12f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:01.818528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d24a88e-d0cd-4989-92b6-263deac0e12f None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:01.824938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:01.825619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:01.825992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:01.826165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] PATH_INFO: `/v3/projects/614cff68b07c457fb1da6fddccf29f41` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:01.843054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] RBAC: Authorizing `identity:delete_project(project_id=614cff68b07c457fb1da6fddccf29f41)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:01.848863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:01.933373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '614cff68b07c457fb1da6fddccf29f41', 'request_id': 'req-dd1e6162-b6aa-41d2-b351-92fc63025589'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:01.944866 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '614cff68b07c457fb1da6fddccf29f41', 'request_id': 'req-dd1e6162-b6aa-41d2-b351-92fc63025589'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:01.952131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '614cff68b07c457fb1da6fddccf29f41', 'request_id': 'req-dd1e6162-b6aa-41d2-b351-92fc63025589'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:01.969514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] The token cache is being invalidate because project 614cff68b07c457fb1da6fddccf29f41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:01.969868 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:01.971234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:01.973334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1040/2098] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:01 2022] DELETE /identity/v3/projects/614cff68b07c457fb1da6fddccf29f41 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:49:01.977497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc62f1d-b4c8-4dc2-ad36-f173d1822a24 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:01.988092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc62f1d-b4c8-4dc2-ad36-f173d1822a24 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:02.060507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:02.061272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:02.061633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:02.061983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] PATH_INFO: `/v3/users/19a1a8c4a4c943b1bbddc3316f79b14b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:02.063287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=19a1a8c4a4c943b1bbddc3316f79b14b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:02.066072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:02.085443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:02.118771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '19a1a8c4a4c943b1bbddc3316f79b14b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:02.119440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '19a1a8c4a4c943b1bbddc3316f79b14b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:02.120282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '19a1a8c4a4c943b1bbddc3316f79b14b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:02.133721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '19a1a8c4a4c943b1bbddc3316f79b14b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:02.141008 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-45a5c93a-a051-4fea-81b5-25ffe405e2ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '19a1a8c4a4c943b1bbddc3316f79b14b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:02.154391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1059/2099] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:01 2022] DELETE /identity/v3/users/19a1a8c4a4c943b1bbddc3316f79b14b => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:49:02.554464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:02.556026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:02.572669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1e6162-b6aa-41d2-b351-92fc63025589 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:02.660067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:02.661036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:02.661533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:02.662133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] PATH_INFO: `/v3/projects/4a6c4b304e7a4b3a89e265d0bb9ffa46` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:02.676094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a6c4b304e7a4b3a89e265d0bb9ffa46)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:02.676094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:02.714960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:02.727938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a6c4b304e7a4b3a89e265d0bb9ffa46', 'request_id': 'req-2f21be04-f057-426c-a07c-f1d1ef92227a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:02.736309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a6c4b304e7a4b3a89e265d0bb9ffa46', 'request_id': 'req-2f21be04-f057-426c-a07c-f1d1ef92227a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:02.743403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a6c4b304e7a4b3a89e265d0bb9ffa46', 'request_id': 'req-2f21be04-f057-426c-a07c-f1d1ef92227a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:02.755215 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] The token cache is being invalidate because project 4a6c4b304e7a4b3a89e265d0bb9ffa46 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:02.755527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:02.756192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2f21be04-f057-426c-a07c-f1d1ef92227a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:02.757713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1041/2100] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:02 2022] DELETE /identity/v3/projects/4a6c4b304e7a4b3a89e265d0bb9ffa46 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:02.763320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b97d0e-264a-49dd-92d8-5a5b8109f142 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:02.763710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b97d0e-264a-49dd-92d8-5a5b8109f142 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:02.764517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b97d0e-264a-49dd-92d8-5a5b8109f142 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:02.784738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-56b97d0e-264a-49dd-92d8-5a5b8109f142 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:02.786365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-56b97d0e-264a-49dd-92d8-5a5b8109f142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:02.846260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1060/2101] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:02.855270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db156a-c923-4830-ad08-393519c81891 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:02.855774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db156a-c923-4830-ad08-393519c81891 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:02.855875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db156a-c923-4830-ad08-393519c81891 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:02.875839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-58db156a-c923-4830-ad08-393519c81891 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:02.877416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-58db156a-c923-4830-ad08-393519c81891 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:02.885473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1042/2102] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:02.889542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b97d0e-264a-49dd-92d8-5a5b8109f142 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:02.899383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b97d0e-264a-49dd-92d8-5a5b8109f142 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:02.904034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73792a2-be6c-47d8-a45d-c2177ff802fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:02.904685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73792a2-be6c-47d8-a45d-c2177ff802fc admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:02.904976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73792a2-be6c-47d8-a45d-c2177ff802fc admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:02.905255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73792a2-be6c-47d8-a45d-c2177ff802fc admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:02.905959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73792a2-be6c-47d8-a45d-c2177ff802fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:49:02.906269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73792a2-be6c-47d8-a45d-c2177ff802fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:02.908558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73792a2-be6c-47d8-a45d-c2177ff802fc admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:02.920861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1061/2103] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:49:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:49:02.925712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db156a-c923-4830-ad08-393519c81891 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:02.934954 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db156a-c923-4830-ad08-393519c81891 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:02.939897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b7196-41b1-41b4-86f5-745f1aeb82b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:02.940692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b7196-41b1-41b4-86f5-745f1aeb82b8 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:02.940981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b7196-41b1-41b4-86f5-745f1aeb82b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:02.941261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5b7196-41b1-41b4-86f5-745f1aeb82b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:02.941935 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5b7196-41b1-41b4-86f5-745f1aeb82b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:02.943858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5b7196-41b1-41b4-86f5-745f1aeb82b8 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:02.972228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1043/2104] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:49:02 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:02.976452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73792a2-be6c-47d8-a45d-c2177ff802fc admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:02.987697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73792a2-be6c-47d8-a45d-c2177ff802fc admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:03.044739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889e5842-9956-4ed4-a72a-c0ac505962d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:03.045639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889e5842-9956-4ed4-a72a-c0ac505962d5 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:03.045954 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889e5842-9956-4ed4-a72a-c0ac505962d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:03.046242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889e5842-9956-4ed4-a72a-c0ac505962d5 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:03.046950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889e5842-9956-4ed4-a72a-c0ac505962d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:03.049156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889e5842-9956-4ed4-a72a-c0ac505962d5 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:03.069997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1062/2105] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:49:02 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:49:03.074175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b7196-41b1-41b4-86f5-745f1aeb82b8 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:03.085544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5b7196-41b1-41b4-86f5-745f1aeb82b8 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:03.090528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5debda2b-4c01-4145-ac25-17b025f92d92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:03.091147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5debda2b-4c01-4145-ac25-17b025f92d92 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:03.091431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5debda2b-4c01-4145-ac25-17b025f92d92 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:03.091709 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5debda2b-4c01-4145-ac25-17b025f92d92 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:03.092470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5debda2b-4c01-4145-ac25-17b025f92d92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:03.094107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5debda2b-4c01-4145-ac25-17b025f92d92 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:03.105345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1044/2106] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:03.109455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889e5842-9956-4ed4-a72a-c0ac505962d5 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:03.120005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889e5842-9956-4ed4-a72a-c0ac505962d5 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:03.125373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0f2362-317e-499b-941f-b2ac36236bd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:03.126008 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0f2362-317e-499b-941f-b2ac36236bd9 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:03.126293 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0f2362-317e-499b-941f-b2ac36236bd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:03.126577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0f2362-317e-499b-941f-b2ac36236bd9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:03.127292 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0f2362-317e-499b-941f-b2ac36236bd9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:03.128933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0f2362-317e-499b-941f-b2ac36236bd9 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:03.134537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:03.143015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1063/2107] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:03 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:03.147138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5debda2b-4c01-4145-ac25-17b025f92d92 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:03.158783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5debda2b-4c01-4145-ac25-17b025f92d92 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:03.163560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a9f2f3-3d1d-45e9-97a2-ca249f8fc1bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:03.164229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a9f2f3-3d1d-45e9-97a2-ca249f8fc1bb admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:03.165002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a9f2f3-3d1d-45e9-97a2-ca249f8fc1bb admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:03.165002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a9f2f3-3d1d-45e9-97a2-ca249f8fc1bb admin admin] PATH_INFO: `/v3/projects/908a58724a604e8f8eb8a4c943fe8b7f/users/e197c5bf77d34c9ba75871d1cd95eab7/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:03.180341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a9f2f3-3d1d-45e9-97a2-ca249f8fc1bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=908a58724a604e8f8eb8a4c943fe8b7f, user_id=e197c5bf77d34c9ba75871d1cd95eab7, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:03.183118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a9f2f3-3d1d-45e9-97a2-ca249f8fc1bb admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:03.196648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1045/2108] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:49:03 2022] PUT /identity/v3/projects/908a58724a604e8f8eb8a4c943fe8b7f/users/e197c5bf77d34c9ba75871d1cd95eab7/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:03.207588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb5d2e-919d-48d2-99fd-8292216cca46 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:03.207879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb5d2e-919d-48d2-99fd-8292216cca46 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:03.208163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43cb5d2e-919d-48d2-99fd-8292216cca46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:03.237113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-43cb5d2e-919d-48d2-99fd-8292216cca46 None None] MFA Rules not processed for user `e197c5bf77d34c9ba75871d1cd95eab7`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:03.238449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-43cb5d2e-919d-48d2-99fd-8292216cca46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:03.312409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1064/2109] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:03 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:03.323790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a9f2f3-3d1d-45e9-97a2-ca249f8fc1bb admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:03.325665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e1a9f2f3-3d1d-45e9-97a2-ca249f8fc1bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:03.403853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a9f2f3-3d1d-45e9-97a2-ca249f8fc1bb admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:03.407547 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01aa0d3c-7b84-4725-9b88-c231a75c9073 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3749dc021aac478f88072ced48f4cee4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:03.408150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01aa0d3c-7b84-4725-9b88-c231a75c9073 service glance] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:03.408435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01aa0d3c-7b84-4725-9b88-c231a75c9073 service glance] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:03.408742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01aa0d3c-7b84-4725-9b88-c231a75c9073 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:03.417012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01aa0d3c-7b84-4725-9b88-c231a75c9073 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:03.418474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01aa0d3c-7b84-4725-9b88-c231a75c9073 service glance] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:03.427396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1046/2110] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:03 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:06.305127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cb5d2e-919d-48d2-99fd-8292216cca46 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:06.307161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-43cb5d2e-919d-48d2-99fd-8292216cca46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:06.379328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43cb5d2e-919d-48d2-99fd-8292216cca46 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:06.383865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:06.384512 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:06.384835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:06.385115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] PATH_INFO: `/v3/users/956854812e1e4010aeeb064237bdcf0c` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:06.386287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] RBAC: Authorizing `identity:delete_user(user_id=956854812e1e4010aeeb064237bdcf0c)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:06.388241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:06.441355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '956854812e1e4010aeeb064237bdcf0c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:06.442051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '956854812e1e4010aeeb064237bdcf0c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:06.442683 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '956854812e1e4010aeeb064237bdcf0c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:06.455994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '956854812e1e4010aeeb064237bdcf0c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:06.464075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '956854812e1e4010aeeb064237bdcf0c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:06.478386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1065/2111] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:06 2022] DELETE /identity/v3/users/956854812e1e4010aeeb064237bdcf0c => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:06.482774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ea849-7fd7-49a9-b459-fe253a2fef38 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:06.483253 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ea849-7fd7-49a9-b459-fe253a2fef38 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:06.483482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9ea849-7fd7-49a9-b459-fe253a2fef38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:06.503803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-9f9ea849-7fd7-49a9-b459-fe253a2fef38 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:06.505134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9f9ea849-7fd7-49a9-b459-fe253a2fef38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:06.547574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:06.573127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1047/2112] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:06.583364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:06.584857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:06.598464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:06.648646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfa4212-e2e3-4e10-9ea8-4fb794528191 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:06.652314 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f426f179-a2bf-419b-9081-534cc1fd2237 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:06.652971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f426f179-a2bf-419b-9081-534cc1fd2237 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:06.653256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f426f179-a2bf-419b-9081-534cc1fd2237 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:06.653535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f426f179-a2bf-419b-9081-534cc1fd2237 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:06.664501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f426f179-a2bf-419b-9081-534cc1fd2237 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:06.666112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f426f179-a2bf-419b-9081-534cc1fd2237 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:06.676883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1066/2113] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:49:07.047869 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ea849-7fd7-49a9-b459-fe253a2fef38 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:07.049808 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9f9ea849-7fd7-49a9-b459-fe253a2fef38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:07.072467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9ea849-7fd7-49a9-b459-fe253a2fef38 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:07.072467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:07.072467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:07.072467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:07.072835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] PATH_INFO: `/v3/projects/301f15b6723146d99b477817ffd324a1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:07.072835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=301f15b6723146d99b477817ffd324a1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:07.072981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:07.122693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '301f15b6723146d99b477817ffd324a1', 'request_id': 'req-0f834852-166d-4903-aa42-64add737c7cb'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.130957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '301f15b6723146d99b477817ffd324a1', 'request_id': 'req-0f834852-166d-4903-aa42-64add737c7cb'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.137279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '301f15b6723146d99b477817ffd324a1', 'request_id': 'req-0f834852-166d-4903-aa42-64add737c7cb'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.148932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] The token cache is being invalidate because project 301f15b6723146d99b477817ffd324a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:07.149228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.149883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.151399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1048/2114] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:07 2022] DELETE /identity/v3/projects/301f15b6723146d99b477817ffd324a1 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:07.155359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f426f179-a2bf-419b-9081-534cc1fd2237 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:07.165852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f426f179-a2bf-419b-9081-534cc1fd2237 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:07.223882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:07.224523 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:07.224846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:07.225128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] PATH_INFO: `/v3/users/9cc73e663e2a41328342ff95a82c781d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:07.226301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9cc73e663e2a41328342ff95a82c781d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:07.228228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:07.276543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9cc73e663e2a41328342ff95a82c781d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.277052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9cc73e663e2a41328342ff95a82c781d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.277691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9cc73e663e2a41328342ff95a82c781d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.289881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9cc73e663e2a41328342ff95a82c781d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.297886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9cc73e663e2a41328342ff95a82c781d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.313421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1067/2115] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:07 2022] DELETE /identity/v3/users/9cc73e663e2a41328342ff95a82c781d => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:07.774737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:07.776340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:07.792148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f834852-166d-4903-aa42-64add737c7cb admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:07.855953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:07.856626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:07.856915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:07.857225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] PATH_INFO: `/v3/projects/4d1e691ae5274c78be10dbebdb9d26a0` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:07.858396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d1e691ae5274c78be10dbebdb9d26a0)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:07.860322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:07.912895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d1e691ae5274c78be10dbebdb9d26a0', 'request_id': 'req-023eac72-c83f-4151-bd80-27e016d3e600'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.920679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d1e691ae5274c78be10dbebdb9d26a0', 'request_id': 'req-023eac72-c83f-4151-bd80-27e016d3e600'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.930765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d1e691ae5274c78be10dbebdb9d26a0', 'request_id': 'req-023eac72-c83f-4151-bd80-27e016d3e600'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.932836 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:07.944040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] The token cache is being invalidate because project 4d1e691ae5274c78be10dbebdb9d26a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:07.944345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.945300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:07.946958 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1049/2116] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:07 2022] DELETE /identity/v3/projects/4d1e691ae5274c78be10dbebdb9d26a0 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:07.950998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:07.952510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:07.965580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30a1558-3a14-41d8-995f-8e6f8bc8de2b admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:07.997366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:08.030465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:08.031124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:08.031418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:08.031706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] PATH_INFO: `/v3/users/ca0933cf62df4f30a2d127a834fee2fc` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:08.033565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca0933cf62df4f30a2d127a834fee2fc)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:08.035803 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:08.085113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca0933cf62df4f30a2d127a834fee2fc'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:08.085787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca0933cf62df4f30a2d127a834fee2fc'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:08.086405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca0933cf62df4f30a2d127a834fee2fc'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:08.100589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca0933cf62df4f30a2d127a834fee2fc'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:08.107443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f96ec9-8ffd-45e0-b0a1-19e6bdf7422f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca0933cf62df4f30a2d127a834fee2fc'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:08.121952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1068/2117] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:07 2022] DELETE /identity/v3/users/ca0933cf62df4f30a2d127a834fee2fc => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:49:08.405709 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:08.407319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:08.427211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023eac72-c83f-4151-bd80-27e016d3e600 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:08.487916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:08.488564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:08.488891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:08.489170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] PATH_INFO: `/v3/projects/d21b087bf09443bf8d13154cfd339983` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:08.490337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d21b087bf09443bf8d13154cfd339983)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:08.492251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:08.543536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd21b087bf09443bf8d13154cfd339983', 'request_id': 'req-27ea1cbf-3d43-4512-b5ca-66e25c95b372'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:08.551770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd21b087bf09443bf8d13154cfd339983', 'request_id': 'req-27ea1cbf-3d43-4512-b5ca-66e25c95b372'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:08.558416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd21b087bf09443bf8d13154cfd339983', 'request_id': 'req-27ea1cbf-3d43-4512-b5ca-66e25c95b372'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:08.574376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] The token cache is being invalidate because project d21b087bf09443bf8d13154cfd339983 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:08.574665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:08.575326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-27ea1cbf-3d43-4512-b5ca-66e25c95b372 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:08.576987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1050/2118] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:08 2022] DELETE /identity/v3/projects/d21b087bf09443bf8d13154cfd339983 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:08.582807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c63857-6367-4f35-be4f-642d999bd93b None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:08.583275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c63857-6367-4f35-be4f-642d999bd93b None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:08.583383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c63857-6367-4f35-be4f-642d999bd93b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:08.606537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-56c63857-6367-4f35-be4f-642d999bd93b None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:08.607669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-56c63857-6367-4f35-be4f-642d999bd93b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:08.677170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1069/2119] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:08.684694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d93cca-55fa-41bf-8659-d21a239ac0cb None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:08.685073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d93cca-55fa-41bf-8659-d21a239ac0cb None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:08.685412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d93cca-55fa-41bf-8659-d21a239ac0cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:08.706863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-95d93cca-55fa-41bf-8659-d21a239ac0cb None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:08.708092 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-95d93cca-55fa-41bf-8659-d21a239ac0cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:08.716168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1051/2120] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:08.720531 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c63857-6367-4f35-be4f-642d999bd93b None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:08.730432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c63857-6367-4f35-be4f-642d999bd93b None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:08.735215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77576a95-bf16-4711-878a-fb068c35bdc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:08.735884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77576a95-bf16-4711-878a-fb068c35bdc1 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:08.736233 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77576a95-bf16-4711-878a-fb068c35bdc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:08.736517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77576a95-bf16-4711-878a-fb068c35bdc1 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:08.737268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77576a95-bf16-4711-878a-fb068c35bdc1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:49:08.737583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77576a95-bf16-4711-878a-fb068c35bdc1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:08.739215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77576a95-bf16-4711-878a-fb068c35bdc1 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:08.751822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1070/2121] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:49:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:49:08.756210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d93cca-55fa-41bf-8659-d21a239ac0cb None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:08.765522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d93cca-55fa-41bf-8659-d21a239ac0cb None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:08.770949 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81379472-ba57-4d73-87bf-55f395c71885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:08.771562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81379472-ba57-4d73-87bf-55f395c71885 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:08.771850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81379472-ba57-4d73-87bf-55f395c71885 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:08.772132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81379472-ba57-4d73-87bf-55f395c71885 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:08.772850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81379472-ba57-4d73-87bf-55f395c71885 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:08.774901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81379472-ba57-4d73-87bf-55f395c71885 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:08.807369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1052/2122] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:49:08 2022] POST /identity/v3/projects => generated 385 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:08.808980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77576a95-bf16-4711-878a-fb068c35bdc1 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:08.820208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77576a95-bf16-4711-878a-fb068c35bdc1 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:08.859898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:08.885581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e215e335-4d36-4b5d-bd89-35d2e229142f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:08.886209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e215e335-4d36-4b5d-bd89-35d2e229142f admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:08.886579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e215e335-4d36-4b5d-bd89-35d2e229142f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:08.886766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e215e335-4d36-4b5d-bd89-35d2e229142f admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:08.887432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e215e335-4d36-4b5d-bd89-35d2e229142f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:08.889376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e215e335-4d36-4b5d-bd89-35d2e229142f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:08.915962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1071/2123] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:49:08 2022] POST /identity/v3/users => generated 293 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:08.917215 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81379472-ba57-4d73-87bf-55f395c71885 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:08.928658 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81379472-ba57-4d73-87bf-55f395c71885 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:08.933638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0083c-ba66-4731-93d3-5b119c2286a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:08.934312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0083c-ba66-4731-93d3-5b119c2286a6 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:08.934585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0083c-ba66-4731-93d3-5b119c2286a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:08.934864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0083c-ba66-4731-93d3-5b119c2286a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:08.935603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf0083c-ba66-4731-93d3-5b119c2286a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:08.937266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf0083c-ba66-4731-93d3-5b119c2286a6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:08.949012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1053/2124] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:49:08.952591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e215e335-4d36-4b5d-bd89-35d2e229142f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:08.963473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e215e335-4d36-4b5d-bd89-35d2e229142f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:08.968156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0f1586-5938-4aaa-b712-8c8f7d8a81e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:08.968874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f1586-5938-4aaa-b712-8c8f7d8a81e2 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:08.969164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f1586-5938-4aaa-b712-8c8f7d8a81e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:08.969445 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0f1586-5938-4aaa-b712-8c8f7d8a81e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:08.970203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0f1586-5938-4aaa-b712-8c8f7d8a81e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:08.971803 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0f1586-5938-4aaa-b712-8c8f7d8a81e2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:08.982670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1072/2125] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:08.987411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0083c-ba66-4731-93d3-5b119c2286a6 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:08.997959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0083c-ba66-4731-93d3-5b119c2286a6 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:09.003702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b408ce4f-b4f8-4922-8eb1-2fb3bce0687b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:09.004518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b408ce4f-b4f8-4922-8eb1-2fb3bce0687b admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.005544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b408ce4f-b4f8-4922-8eb1-2fb3bce0687b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.005902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b408ce4f-b4f8-4922-8eb1-2fb3bce0687b admin admin] PATH_INFO: `/v3/projects/8b3fd0d6c4da4de3af7a056759d0e04f/users/04a9d02da16447cdba119b464be30d22/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.022856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b408ce4f-b4f8-4922-8eb1-2fb3bce0687b admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b3fd0d6c4da4de3af7a056759d0e04f, user_id=04a9d02da16447cdba119b464be30d22, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:09.025813 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b408ce4f-b4f8-4922-8eb1-2fb3bce0687b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:09.039363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1054/2126] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:49:08 2022] PUT /identity/v3/projects/8b3fd0d6c4da4de3af7a056759d0e04f/users/04a9d02da16447cdba119b464be30d22/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:09.115481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfeac91-8461-4887-87c2-4fe8f29b4597 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.115797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfeac91-8461-4887-87c2-4fe8f29b4597 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.116188 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfeac91-8461-4887-87c2-4fe8f29b4597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.152470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-4cfeac91-8461-4887-87c2-4fe8f29b4597 None None] MFA Rules not processed for user `04a9d02da16447cdba119b464be30d22`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:09.155010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-4cfeac91-8461-4887-87c2-4fe8f29b4597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:09.236674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1073/2127] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:09 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:09.238302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b408ce4f-b4f8-4922-8eb1-2fb3bce0687b admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:09.265074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:09.308116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b408ce4f-b4f8-4922-8eb1-2fb3bce0687b admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:09.312908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba075a2-dd7d-4b9b-b78b-726cb8051894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:09.313551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba075a2-dd7d-4b9b-b78b-726cb8051894 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.314660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba075a2-dd7d-4b9b-b78b-726cb8051894 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.315071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba075a2-dd7d-4b9b-b78b-726cb8051894 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.315778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba075a2-dd7d-4b9b-b78b-726cb8051894 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:09.317714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba075a2-dd7d-4b9b-b78b-726cb8051894 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:09.344153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1055/2128] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:49:09 2022] POST /identity/v3/projects => generated 385 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:09.348890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfeac91-8461-4887-87c2-4fe8f29b4597 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:09.365935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfeac91-8461-4887-87c2-4fe8f29b4597 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:09.422322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a49053-12c4-4793-9596-ad85393f0c74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:09.422944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a49053-12c4-4793-9596-ad85393f0c74 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.423851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a49053-12c4-4793-9596-ad85393f0c74 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.424110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a49053-12c4-4793-9596-ad85393f0c74 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.424814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a49053-12c4-4793-9596-ad85393f0c74 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:09.426922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a49053-12c4-4793-9596-ad85393f0c74 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:09.447425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1074/2129] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:49:09 2022] POST /identity/v3/users => generated 293 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:09.452006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba075a2-dd7d-4b9b-b78b-726cb8051894 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:09.463052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba075a2-dd7d-4b9b-b78b-726cb8051894 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:09.467758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9293f3ed-a55e-4b7e-853f-90519d7ceb5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:09.468374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9293f3ed-a55e-4b7e-853f-90519d7ceb5b admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.468693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9293f3ed-a55e-4b7e-853f-90519d7ceb5b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.468977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9293f3ed-a55e-4b7e-853f-90519d7ceb5b admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.469708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9293f3ed-a55e-4b7e-853f-90519d7ceb5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:09.471977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9293f3ed-a55e-4b7e-853f-90519d7ceb5b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:09.483221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1056/2130] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:09.488050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a49053-12c4-4793-9596-ad85393f0c74 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:09.499155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a49053-12c4-4793-9596-ad85393f0c74 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:09.504299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce99b615-93fc-4d91-b5f8-733ef4218015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:09.504946 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99b615-93fc-4d91-b5f8-733ef4218015 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.505717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99b615-93fc-4d91-b5f8-733ef4218015 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.505717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99b615-93fc-4d91-b5f8-733ef4218015 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.506308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce99b615-93fc-4d91-b5f8-733ef4218015 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:09.507918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce99b615-93fc-4d91-b5f8-733ef4218015 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:09.518794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1075/2131] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:49:09.522988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9293f3ed-a55e-4b7e-853f-90519d7ceb5b admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:09.534380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9293f3ed-a55e-4b7e-853f-90519d7ceb5b admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:09.544991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f473829-9340-4a60-b8a3-3eeb3b282213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:09.546347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f473829-9340-4a60-b8a3-3eeb3b282213 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.546831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f473829-9340-4a60-b8a3-3eeb3b282213 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.547307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f473829-9340-4a60-b8a3-3eeb3b282213 admin admin] PATH_INFO: `/v3/projects/4c6b868a7d434e9aa19f89070d4ffaca/users/6a922f7e39104f629b393bac9f6c8bf4/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.571854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f473829-9340-4a60-b8a3-3eeb3b282213 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c6b868a7d434e9aa19f89070d4ffaca, user_id=6a922f7e39104f629b393bac9f6c8bf4, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:09.576873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f473829-9340-4a60-b8a3-3eeb3b282213 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:09.600527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1057/2132] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:49:09 2022] PUT /identity/v3/projects/4c6b868a7d434e9aa19f89070d4ffaca/users/6a922f7e39104f629b393bac9f6c8bf4/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:09.611294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970eba17-30c1-4aa5-a8c3-5cde9fe42c5c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.611777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970eba17-30c1-4aa5-a8c3-5cde9fe42c5c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.612068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970eba17-30c1-4aa5-a8c3-5cde9fe42c5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.642682 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-970eba17-30c1-4aa5-a8c3-5cde9fe42c5c None None] MFA Rules not processed for user `6a922f7e39104f629b393bac9f6c8bf4`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:09.643889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-970eba17-30c1-4aa5-a8c3-5cde9fe42c5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:09.656164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:09.712156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1076/2133] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:09 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:09.716656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f473829-9340-4a60-b8a3-3eeb3b282213 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:09.785690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f473829-9340-4a60-b8a3-3eeb3b282213 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:09.790582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90e9188-7779-49ba-8fce-a728f6fe7328 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:09.791176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90e9188-7779-49ba-8fce-a728f6fe7328 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.791462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90e9188-7779-49ba-8fce-a728f6fe7328 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.791793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90e9188-7779-49ba-8fce-a728f6fe7328 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.792461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90e9188-7779-49ba-8fce-a728f6fe7328 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:09.794481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90e9188-7779-49ba-8fce-a728f6fe7328 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:09.826094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1058/2134] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:49:09 2022] POST /identity/v3/projects => generated 383 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:49:09.827513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970eba17-30c1-4aa5-a8c3-5cde9fe42c5c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:09.840427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970eba17-30c1-4aa5-a8c3-5cde9fe42c5c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:09.901261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a2a837-50ba-42c0-af65-00241ba5d402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:09.902033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a2a837-50ba-42c0-af65-00241ba5d402 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.902321 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a2a837-50ba-42c0-af65-00241ba5d402 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.902596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a2a837-50ba-42c0-af65-00241ba5d402 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.903294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a2a837-50ba-42c0-af65-00241ba5d402 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:09.905265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a2a837-50ba-42c0-af65-00241ba5d402 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:09.931935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1077/2135] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:49:09 2022] POST /identity/v3/users => generated 298 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:09.933764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90e9188-7779-49ba-8fce-a728f6fe7328 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:09.946174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90e9188-7779-49ba-8fce-a728f6fe7328 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:09.953072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39c0825-264c-4552-91aa-2e0e5ad6686b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:09.953846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39c0825-264c-4552-91aa-2e0e5ad6686b admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.954265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39c0825-264c-4552-91aa-2e0e5ad6686b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.955340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39c0825-264c-4552-91aa-2e0e5ad6686b admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:09.956019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39c0825-264c-4552-91aa-2e0e5ad6686b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:09.957775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39c0825-264c-4552-91aa-2e0e5ad6686b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:09.971034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1059/2136] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:09 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:09.975302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a2a837-50ba-42c0-af65-00241ba5d402 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:09.986378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a2a837-50ba-42c0-af65-00241ba5d402 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:09.991244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cf4761-d60b-4f48-8abe-dcba320a3732 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:09.992013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cf4761-d60b-4f48-8abe-dcba320a3732 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:09.992378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cf4761-d60b-4f48-8abe-dcba320a3732 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:09.992758 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cf4761-d60b-4f48-8abe-dcba320a3732 admin admin] PATH_INFO: `/v3/projects/0f0569bfe5194719a6dc1ff40c03c877/users/aec8a599d57e436a80e1e15f6fc29f4f/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:10.013672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cf4761-d60b-4f48-8abe-dcba320a3732 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f0569bfe5194719a6dc1ff40c03c877, user_id=aec8a599d57e436a80e1e15f6fc29f4f, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:10.017908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cf4761-d60b-4f48-8abe-dcba320a3732 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:10.036151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1078/2137] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:49:09 2022] PUT /identity/v3/projects/0f0569bfe5194719a6dc1ff40c03c877/users/aec8a599d57e436a80e1e15f6fc29f4f/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:49:10.040467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39c0825-264c-4552-91aa-2e0e5ad6686b admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:10.053509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39c0825-264c-4552-91aa-2e0e5ad6686b admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:10.112491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:10.132952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0646f091-cf66-4772-8bb3-b2258a2bce35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:10.132952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0646f091-cf66-4772-8bb3-b2258a2bce35 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:10.132952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0646f091-cf66-4772-8bb3-b2258a2bce35 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:10.132952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0646f091-cf66-4772-8bb3-b2258a2bce35 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:10.133428 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0646f091-cf66-4772-8bb3-b2258a2bce35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:10.133428 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0646f091-cf66-4772-8bb3-b2258a2bce35 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:10.145389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1060/2138] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:10 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:49:10.149384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cf4761-d60b-4f48-8abe-dcba320a3732 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:10.163423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cf4761-d60b-4f48-8abe-dcba320a3732 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:10.169294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98e7154-9575-4038-be19-9ca12ea2f85d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:10.170091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98e7154-9575-4038-be19-9ca12ea2f85d admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:10.170674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98e7154-9575-4038-be19-9ca12ea2f85d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:10.170954 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98e7154-9575-4038-be19-9ca12ea2f85d admin admin] PATH_INFO: `/v3/projects/0f0569bfe5194719a6dc1ff40c03c877/users/aec8a599d57e436a80e1e15f6fc29f4f/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:10.172955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98e7154-9575-4038-be19-9ca12ea2f85d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f0569bfe5194719a6dc1ff40c03c877, user_id=aec8a599d57e436a80e1e15f6fc29f4f, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:10.175932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98e7154-9575-4038-be19-9ca12ea2f85d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:10.186757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1079/2139] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:49:10 2022] PUT /identity/v3/projects/0f0569bfe5194719a6dc1ff40c03c877/users/aec8a599d57e436a80e1e15f6fc29f4f/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:10.198706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6e98e-0d9d-4468-bc05-f1b324671495 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:10.198881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6e98e-0d9d-4468-bc05-f1b324671495 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:10.199170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd6e98e-0d9d-4468-bc05-f1b324671495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:10.229540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-3dd6e98e-0d9d-4468-bc05-f1b324671495 None None] MFA Rules not processed for user `aec8a599d57e436a80e1e15f6fc29f4f`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:10.230904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3dd6e98e-0d9d-4468-bc05-f1b324671495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:10.308306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1061/2140] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:10 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:10.320356 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98e7154-9575-4038-be19-9ca12ea2f85d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:10.322535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f98e7154-9575-4038-be19-9ca12ea2f85d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:10.383331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98e7154-9575-4038-be19-9ca12ea2f85d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:10.387421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8270ef9-3307-487c-bf33-1560ab7ec542 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:10.388092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8270ef9-3307-487c-bf33-1560ab7ec542 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:10.388522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8270ef9-3307-487c-bf33-1560ab7ec542 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:10.388856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8270ef9-3307-487c-bf33-1560ab7ec542 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:10.409475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:10.450276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8270ef9-3307-487c-bf33-1560ab7ec542 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:10.451785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8270ef9-3307-487c-bf33-1560ab7ec542 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:10.462643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1080/2141] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:10 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:49:11.977026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd6e98e-0d9d-4468-bc05-f1b324671495 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:11.986847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd6e98e-0d9d-4468-bc05-f1b324671495 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:11.990520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a5ea4a-3c54-4270-818c-8cb9a81bbf2a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:11.991136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a5ea4a-3c54-4270-818c-8cb9a81bbf2a service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:11.991424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a5ea4a-3c54-4270-818c-8cb9a81bbf2a service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:11.991706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a5ea4a-3c54-4270-818c-8cb9a81bbf2a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:12.051100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a5ea4a-3c54-4270-818c-8cb9a81bbf2a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:12.052463 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a5ea4a-3c54-4270-818c-8cb9a81bbf2a service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:12.061969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1062/2142] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:11 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:12.718750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8270ef9-3307-487c-bf33-1560ab7ec542 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:12.728999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8270ef9-3307-487c-bf33-1560ab7ec542 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:12.733896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b65fd40-93a2-4d73-b2f3-e18735ea3dc1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:12.734338 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b65fd40-93a2-4d73-b2f3-e18735ea3dc1 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:12.734619 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b65fd40-93a2-4d73-b2f3-e18735ea3dc1 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:12.734893 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b65fd40-93a2-4d73-b2f3-e18735ea3dc1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:12.745131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b65fd40-93a2-4d73-b2f3-e18735ea3dc1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:12.746584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b65fd40-93a2-4d73-b2f3-e18735ea3dc1 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:12.757716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1081/2143] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:12 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:14.831098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f74a25-2db7-4e88-b28f-0893104b5907 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:14.831634 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f74a25-2db7-4e88-b28f-0893104b5907 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:14.831712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f74a25-2db7-4e88-b28f-0893104b5907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:14.852985 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-43f74a25-2db7-4e88-b28f-0893104b5907 None None] MFA Rules not processed for user `87f97b8b406e442b93cbdda090d7e9ff`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:14.854225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-43f74a25-2db7-4e88-b28f-0893104b5907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:14.918929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1063/2144] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:14 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:16.683199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b65fd40-93a2-4d73-b2f3-e18735ea3dc1 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:16.776056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b65fd40-93a2-4d73-b2f3-e18735ea3dc1 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:16.787586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:16.788317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:16.788708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:16.788993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] PATH_INFO: `/v3/users/04a9d02da16447cdba119b464be30d22` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:16.790656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] RBAC: Authorizing `identity:delete_user(user_id=04a9d02da16447cdba119b464be30d22)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:16.794363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:16.883767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04a9d02da16447cdba119b464be30d22'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:16.883767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04a9d02da16447cdba119b464be30d22'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:16.883767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04a9d02da16447cdba119b464be30d22'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:16.905996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04a9d02da16447cdba119b464be30d22'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:16.915036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04a9d02da16447cdba119b464be30d22'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:16.948960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1082/2145] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:16 2022] DELETE /identity/v3/users/04a9d02da16447cdba119b464be30d22 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:16.954750 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdb430b-58b7-4c69-9483-1ee29dfa1998 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:16.955309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdb430b-58b7-4c69-9483-1ee29dfa1998 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:16.955309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdb430b-58b7-4c69-9483-1ee29dfa1998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:16.979029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-ccdb430b-58b7-4c69-9483-1ee29dfa1998 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:16.981187 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ccdb430b-58b7-4c69-9483-1ee29dfa1998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:16.989137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:17.074702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1064/2146] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:17.095881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:17.096583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:17.149888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:17.189349 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da037ed-f946-4250-a204-f826da268654 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:17.193187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b0a53-c99a-4ce8-aaf5-c445a9e86a11 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:17.194963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b0a53-c99a-4ce8-aaf5-c445a9e86a11 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:17.195262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b0a53-c99a-4ce8-aaf5-c445a9e86a11 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:17.195547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692b0a53-c99a-4ce8-aaf5-c445a9e86a11 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:17.207362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692b0a53-c99a-4ce8-aaf5-c445a9e86a11 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:17.208848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692b0a53-c99a-4ce8-aaf5-c445a9e86a11 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:17.222960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1083/2147] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:18.013214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdb430b-58b7-4c69-9483-1ee29dfa1998 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:18.017585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ccdb430b-58b7-4c69-9483-1ee29dfa1998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:18.040994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdb430b-58b7-4c69-9483-1ee29dfa1998 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:18.048797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:18.049855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:18.050640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:18.051466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] PATH_INFO: `/v3/projects/8b3fd0d6c4da4de3af7a056759d0e04f` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:18.055559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b3fd0d6c4da4de3af7a056759d0e04f)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:18.059892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:18.143279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b3fd0d6c4da4de3af7a056759d0e04f', 'request_id': 'req-e0bb89af-1a72-48b8-be0c-f833bb995754'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:18.155498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b3fd0d6c4da4de3af7a056759d0e04f', 'request_id': 'req-e0bb89af-1a72-48b8-be0c-f833bb995754'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:18.165085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b3fd0d6c4da4de3af7a056759d0e04f', 'request_id': 'req-e0bb89af-1a72-48b8-be0c-f833bb995754'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:18.181434 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] The token cache is being invalidate because project 8b3fd0d6c4da4de3af7a056759d0e04f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:18.181726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:18.182823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:18.185074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1065/2148] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:18 2022] DELETE /identity/v3/projects/8b3fd0d6c4da4de3af7a056759d0e04f => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:18.189199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b0a53-c99a-4ce8-aaf5-c445a9e86a11 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:18.212033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692b0a53-c99a-4ce8-aaf5-c445a9e86a11 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:18.294808 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:18.296043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:18.296535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:18.296976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] PATH_INFO: `/v3/users/6a922f7e39104f629b393bac9f6c8bf4` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:18.298287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a922f7e39104f629b393bac9f6c8bf4)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:18.300219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:18.378914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a922f7e39104f629b393bac9f6c8bf4'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:18.379909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a922f7e39104f629b393bac9f6c8bf4'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:18.380745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a922f7e39104f629b393bac9f6c8bf4'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:18.404789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a922f7e39104f629b393bac9f6c8bf4'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:18.425365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a922f7e39104f629b393bac9f6c8bf4'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:18.445038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1084/2149] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:18 2022] DELETE /identity/v3/users/6a922f7e39104f629b393bac9f6c8bf4 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:19.092355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:19.095159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:19.109169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bb89af-1a72-48b8-be0c-f833bb995754 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:19.182116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:19.183154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:19.183644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:19.184168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] PATH_INFO: `/v3/projects/4c6b868a7d434e9aa19f89070d4ffaca` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:19.185639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c6b868a7d434e9aa19f89070d4ffaca)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:19.187816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:19.241642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:19.258010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c6b868a7d434e9aa19f89070d4ffaca', 'request_id': 'req-0959362e-010c-494a-861a-4c98b28023b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:19.273055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c6b868a7d434e9aa19f89070d4ffaca', 'request_id': 'req-0959362e-010c-494a-861a-4c98b28023b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:19.279994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c6b868a7d434e9aa19f89070d4ffaca', 'request_id': 'req-0959362e-010c-494a-861a-4c98b28023b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:19.296743 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] The token cache is being invalidate because project 4c6b868a7d434e9aa19f89070d4ffaca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:19.297183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:19.298766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:19.300346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1066/2150] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:19 2022] DELETE /identity/v3/projects/4c6b868a7d434e9aa19f89070d4ffaca => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:19.304412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:19.306538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:19.333380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28f805a-4a62-4a59-90e2-67a45a6c2e59 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:19.407493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:19.429369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:19.430224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:19.430627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:19.431094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] PATH_INFO: `/v3/users/aec8a599d57e436a80e1e15f6fc29f4f` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:19.433528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aec8a599d57e436a80e1e15f6fc29f4f)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:19.436444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:19.494443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aec8a599d57e436a80e1e15f6fc29f4f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:19.495204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aec8a599d57e436a80e1e15f6fc29f4f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:19.496036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aec8a599d57e436a80e1e15f6fc29f4f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:19.510886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aec8a599d57e436a80e1e15f6fc29f4f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:19.518759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a2e5ecc2-6f88-4b75-af48-dc0cd0bd6099 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aec8a599d57e436a80e1e15f6fc29f4f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:19.533473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1085/2151] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:19 2022] DELETE /identity/v3/users/aec8a599d57e436a80e1e15f6fc29f4f => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:20.029256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:20.031068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:20.046930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0959362e-010c-494a-861a-4c98b28023b8 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:20.118058 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:20.118736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:20.119013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:20.119295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] PATH_INFO: `/v3/projects/0f0569bfe5194719a6dc1ff40c03c877` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:20.120479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f0569bfe5194719a6dc1ff40c03c877)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:20.122528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:20.179331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f0569bfe5194719a6dc1ff40c03c877', 'request_id': 'req-452abcbb-b913-4acc-8051-5f99ff5c20b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:20.190371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f0569bfe5194719a6dc1ff40c03c877', 'request_id': 'req-452abcbb-b913-4acc-8051-5f99ff5c20b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:20.196854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f0569bfe5194719a6dc1ff40c03c877', 'request_id': 'req-452abcbb-b913-4acc-8051-5f99ff5c20b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:20.209545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] The token cache is being invalidate because project 0f0569bfe5194719a6dc1ff40c03c877 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:20.209837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:20.210507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-452abcbb-b913-4acc-8051-5f99ff5c20b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:20.212097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1067/2152] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:20 2022] DELETE /identity/v3/projects/0f0569bfe5194719a6dc1ff40c03c877 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:20.217099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff8911-00a6-421f-bcb6-007cef2a6611 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:20.217729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff8911-00a6-421f-bcb6-007cef2a6611 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:20.217729 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff8911-00a6-421f-bcb6-007cef2a6611 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:20.246811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-dfff8911-00a6-421f-bcb6-007cef2a6611 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:20.250349 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-dfff8911-00a6-421f-bcb6-007cef2a6611 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:20.339458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1086/2153] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:20.347230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7fa664-68ee-402c-8051-0a733bd8fa42 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:20.347545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7fa664-68ee-402c-8051-0a733bd8fa42 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:20.348155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7fa664-68ee-402c-8051-0a733bd8fa42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:20.376769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-ca7fa664-68ee-402c-8051-0a733bd8fa42 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:20.378081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ca7fa664-68ee-402c-8051-0a733bd8fa42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:20.386845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1068/2154] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:20.391173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfff8911-00a6-421f-bcb6-007cef2a6611 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:20.402358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfff8911-00a6-421f-bcb6-007cef2a6611 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:20.407626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98b28af-360f-4567-9023-b55b9c9bc164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:20.408319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98b28af-360f-4567-9023-b55b9c9bc164 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:20.408624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98b28af-360f-4567-9023-b55b9c9bc164 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:20.408928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98b28af-360f-4567-9023-b55b9c9bc164 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:20.409644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98b28af-360f-4567-9023-b55b9c9bc164 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:49:20.409958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98b28af-360f-4567-9023-b55b9c9bc164 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:20.411700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98b28af-360f-4567-9023-b55b9c9bc164 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:20.428448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1087/2155] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:49:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:49:20.429557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7fa664-68ee-402c-8051-0a733bd8fa42 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:20.440142 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7fa664-68ee-402c-8051-0a733bd8fa42 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:20.445822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59724eca-f22d-450f-8a05-06f68a69266d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:20.446561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59724eca-f22d-450f-8a05-06f68a69266d admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:20.447129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59724eca-f22d-450f-8a05-06f68a69266d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:20.447713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59724eca-f22d-450f-8a05-06f68a69266d admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:20.448695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59724eca-f22d-450f-8a05-06f68a69266d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:20.451411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59724eca-f22d-450f-8a05-06f68a69266d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:20.484640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1069/2156] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:49:20 2022] POST /identity/v3/projects => generated 361 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:49:20.488930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98b28af-360f-4567-9023-b55b9c9bc164 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:20.505072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98b28af-360f-4567-9023-b55b9c9bc164 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:20.574968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:20.587650 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447db485-6568-40c1-8c3f-4c79356cb928 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:20.588854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447db485-6568-40c1-8c3f-4c79356cb928 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:20.589298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447db485-6568-40c1-8c3f-4c79356cb928 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:20.589786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447db485-6568-40c1-8c3f-4c79356cb928 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:20.591041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447db485-6568-40c1-8c3f-4c79356cb928 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:20.593279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447db485-6568-40c1-8c3f-4c79356cb928 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:20.627441 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1088/2157] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:49:20 2022] POST /identity/v3/users => generated 281 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:20.628637 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59724eca-f22d-450f-8a05-06f68a69266d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:20.644929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59724eca-f22d-450f-8a05-06f68a69266d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:20.653703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b38ce38-cccd-4998-8c4a-b0e8312cc916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:20.653703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b38ce38-cccd-4998-8c4a-b0e8312cc916 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:20.653703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b38ce38-cccd-4998-8c4a-b0e8312cc916 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:20.653703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b38ce38-cccd-4998-8c4a-b0e8312cc916 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:20.654164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b38ce38-cccd-4998-8c4a-b0e8312cc916 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:20.654164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b38ce38-cccd-4998-8c4a-b0e8312cc916 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:20.676116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1070/2158] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:20 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:49:20.679881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447db485-6568-40c1-8c3f-4c79356cb928 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:20.693349 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447db485-6568-40c1-8c3f-4c79356cb928 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:20.699763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9340a28-fd5a-4dd7-9fff-e968c500bf04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:20.701411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9340a28-fd5a-4dd7-9fff-e968c500bf04 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:20.701787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9340a28-fd5a-4dd7-9fff-e968c500bf04 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:20.705000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9340a28-fd5a-4dd7-9fff-e968c500bf04 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:20.705755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9340a28-fd5a-4dd7-9fff-e968c500bf04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:20.707404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9340a28-fd5a-4dd7-9fff-e968c500bf04 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:20.720656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1089/2159] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:20 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:20.724307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b38ce38-cccd-4998-8c4a-b0e8312cc916 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:20.741409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b38ce38-cccd-4998-8c4a-b0e8312cc916 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:20.746613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03cb6c7-40ab-40bc-9251-27746c60d549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:20.747297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03cb6c7-40ab-40bc-9251-27746c60d549 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:20.747796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03cb6c7-40ab-40bc-9251-27746c60d549 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:20.747916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03cb6c7-40ab-40bc-9251-27746c60d549 admin admin] PATH_INFO: `/v3/projects/fb2577b0b5a946a5beee64c611d5b768/users/3bc686e22f604f5ebb144cd0c1c7748c/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:20.764997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:20.769219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03cb6c7-40ab-40bc-9251-27746c60d549 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb2577b0b5a946a5beee64c611d5b768, user_id=3bc686e22f604f5ebb144cd0c1c7748c, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:20.772660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03cb6c7-40ab-40bc-9251-27746c60d549 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:20.798822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1071/2160] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:49:20 2022] PUT /identity/v3/projects/fb2577b0b5a946a5beee64c611d5b768/users/3bc686e22f604f5ebb144cd0c1c7748c/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:49:20.816828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fb992d-9f18-4798-9e07-eb50f0266f57 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:20.817732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fb992d-9f18-4798-9e07-eb50f0266f57 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:20.817732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fb992d-9f18-4798-9e07-eb50f0266f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:20.877180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-66fb992d-9f18-4798-9e07-eb50f0266f57 None None] MFA Rules not processed for user `3bc686e22f604f5ebb144cd0c1c7748c`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:20.879114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-66fb992d-9f18-4798-9e07-eb50f0266f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:21.043340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1090/2161] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:20 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:21.053573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03cb6c7-40ab-40bc-9251-27746c60d549 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:21.058330 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d03cb6c7-40ab-40bc-9251-27746c60d549 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:21.194332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03cb6c7-40ab-40bc-9251-27746c60d549 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:21.202112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6509030-9f06-4860-aed5-aaa9c131271d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:21.203449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6509030-9f06-4860-aed5-aaa9c131271d service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:21.204223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6509030-9f06-4860-aed5-aaa9c131271d service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:21.204976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6509030-9f06-4860-aed5-aaa9c131271d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:21.222071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6509030-9f06-4860-aed5-aaa9c131271d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:21.225361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6509030-9f06-4860-aed5-aaa9c131271d service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:21.243884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1072/2162] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:21 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:21.616459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fb992d-9f18-4798-9e07-eb50f0266f57 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:21.618559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-66fb992d-9f18-4798-9e07-eb50f0266f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:21.717867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fb992d-9f18-4798-9e07-eb50f0266f57 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:21.722932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:21.723574 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:21.723860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:21.724138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] PATH_INFO: `/v3/users/34242a2360f14710af2e8dbe6b704ca1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:21.725315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34242a2360f14710af2e8dbe6b704ca1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:21.727250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:21.773176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34242a2360f14710af2e8dbe6b704ca1'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:21.773829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34242a2360f14710af2e8dbe6b704ca1'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:21.774451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34242a2360f14710af2e8dbe6b704ca1'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:21.786297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34242a2360f14710af2e8dbe6b704ca1'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:21.787478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:21.793828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1d1674a6-b0b1-4dce-8d6b-2428337edca2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34242a2360f14710af2e8dbe6b704ca1'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:21.817879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1091/2163] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:21 2022] DELETE /identity/v3/users/34242a2360f14710af2e8dbe6b704ca1 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:21.995051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6509030-9f06-4860-aed5-aaa9c131271d service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:22.006889 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6509030-9f06-4860-aed5-aaa9c131271d service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:22.065846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:22.066491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.066781 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.067065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] PATH_INFO: `/v3/projects/77688c4faed945849f5649c08cc580c2` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.068235 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=77688c4faed945849f5649c08cc580c2)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:22.070185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:22.118478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77688c4faed945849f5649c08cc580c2', 'request_id': 'req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:22.125138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77688c4faed945849f5649c08cc580c2', 'request_id': 'req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:22.130976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77688c4faed945849f5649c08cc580c2', 'request_id': 'req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:22.141764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] The token cache is being invalidate because project 77688c4faed945849f5649c08cc580c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:22.142049 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:22.142682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-0120b5b9-d2cb-4d3a-9ce3-45bb05abce1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:22.144187 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1073/2164] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:21 2022] DELETE /identity/v3/projects/77688c4faed945849f5649c08cc580c2 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:22.149343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860dbb5c-0b98-4d94-be78-563884158518 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.149642 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860dbb5c-0b98-4d94-be78-563884158518 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.149932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860dbb5c-0b98-4d94-be78-563884158518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.170443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-860dbb5c-0b98-4d94-be78-563884158518 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:22.171668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-860dbb5c-0b98-4d94-be78-563884158518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:22.235804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1092/2165] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:22.242434 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486e896-4d4e-4900-8c8b-d868ef1dfc3c None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.243065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486e896-4d4e-4900-8c8b-d868ef1dfc3c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.243065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9486e896-4d4e-4900-8c8b-d868ef1dfc3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.263224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-9486e896-4d4e-4900-8c8b-d868ef1dfc3c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:22.264496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9486e896-4d4e-4900-8c8b-d868ef1dfc3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:22.276721 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1074/2166] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:22.281065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860dbb5c-0b98-4d94-be78-563884158518 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:22.292388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860dbb5c-0b98-4d94-be78-563884158518 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:22.297699 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf52649-3b67-4549-a7e0-99da4e7a0251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:22.298804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf52649-3b67-4549-a7e0-99da4e7a0251 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.299162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf52649-3b67-4549-a7e0-99da4e7a0251 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.299518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf52649-3b67-4549-a7e0-99da4e7a0251 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.300305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf52649-3b67-4549-a7e0-99da4e7a0251 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:49:22.300732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf52649-3b67-4549-a7e0-99da4e7a0251 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:22.302443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf52649-3b67-4549-a7e0-99da4e7a0251 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:22.318945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1093/2167] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:49:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:49:22.325013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9486e896-4d4e-4900-8c8b-d868ef1dfc3c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:22.336406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9486e896-4d4e-4900-8c8b-d868ef1dfc3c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:22.341531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bbd95b-2e24-4d42-9df4-340deadd68e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:22.342168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bbd95b-2e24-4d42-9df4-340deadd68e0 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.342452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bbd95b-2e24-4d42-9df4-340deadd68e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.342893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75bbd95b-2e24-4d42-9df4-340deadd68e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.343728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bbd95b-2e24-4d42-9df4-340deadd68e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:22.347115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75bbd95b-2e24-4d42-9df4-340deadd68e0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:22.365501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:22.379526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1075/2168] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:49:22 2022] POST /identity/v3/projects => generated 389 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:22.384203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf52649-3b67-4549-a7e0-99da4e7a0251 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:22.394853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf52649-3b67-4549-a7e0-99da4e7a0251 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:22.450711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244383b2-f9fe-40d0-82d8-653d49df25a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:22.451332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244383b2-f9fe-40d0-82d8-653d49df25a5 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.451621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244383b2-f9fe-40d0-82d8-653d49df25a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.451904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244383b2-f9fe-40d0-82d8-653d49df25a5 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.452712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244383b2-f9fe-40d0-82d8-653d49df25a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:22.455274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244383b2-f9fe-40d0-82d8-653d49df25a5 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:22.476427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1094/2169] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:49:22 2022] POST /identity/v3/users => generated 295 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:49:22.480554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bbd95b-2e24-4d42-9df4-340deadd68e0 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:22.491017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75bbd95b-2e24-4d42-9df4-340deadd68e0 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:22.495738 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107d935a-c3bc-4b78-87e8-ba117b5ca530 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:22.496293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107d935a-c3bc-4b78-87e8-ba117b5ca530 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.496589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107d935a-c3bc-4b78-87e8-ba117b5ca530 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.496900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107d935a-c3bc-4b78-87e8-ba117b5ca530 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.497601 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107d935a-c3bc-4b78-87e8-ba117b5ca530 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:22.499200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107d935a-c3bc-4b78-87e8-ba117b5ca530 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:22.509135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1076/2170] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:22 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:22.512923 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244383b2-f9fe-40d0-82d8-653d49df25a5 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:22.523367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244383b2-f9fe-40d0-82d8-653d49df25a5 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:22.527903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e48963b8-a455-49fc-b894-90062eb75ff5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:22.528499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48963b8-a455-49fc-b894-90062eb75ff5 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.528828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48963b8-a455-49fc-b894-90062eb75ff5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.529108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e48963b8-a455-49fc-b894-90062eb75ff5 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.529797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48963b8-a455-49fc-b894-90062eb75ff5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:22.531380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e48963b8-a455-49fc-b894-90062eb75ff5 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:22.542802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1095/2171] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:22.547247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107d935a-c3bc-4b78-87e8-ba117b5ca530 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:22.557751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107d935a-c3bc-4b78-87e8-ba117b5ca530 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:22.562488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe41c2ca-a90f-442b-9270-a846bef0d0f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:22.563227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe41c2ca-a90f-442b-9270-a846bef0d0f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.563499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe41c2ca-a90f-442b-9270-a846bef0d0f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.563811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe41c2ca-a90f-442b-9270-a846bef0d0f9 admin admin] PATH_INFO: `/v3/projects/f8e9e21d168140e99ffed71146e35fc4/users/a1e1c24849d7480b8ac804a7e3d2ab5a/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.578058 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe41c2ca-a90f-442b-9270-a846bef0d0f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f8e9e21d168140e99ffed71146e35fc4, user_id=a1e1c24849d7480b8ac804a7e3d2ab5a, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:22.581316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe41c2ca-a90f-442b-9270-a846bef0d0f9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:22.593960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1077/2172] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:49:22 2022] PUT /identity/v3/projects/f8e9e21d168140e99ffed71146e35fc4/users/a1e1c24849d7480b8ac804a7e3d2ab5a/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:22.599505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbdd15c-ac54-454f-93a3-82ed3f7ea964 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.599799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbdd15c-ac54-454f-93a3-82ed3f7ea964 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.600080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbdd15c-ac54-454f-93a3-82ed3f7ea964 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.618661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-fbbdd15c-ac54-454f-93a3-82ed3f7ea964 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:22.619879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fbbdd15c-ac54-454f-93a3-82ed3f7ea964 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:22.668307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:22.686290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1096/2173] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:22.701574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe41c2ca-a90f-442b-9270-a846bef0d0f9 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:22.704287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fe41c2ca-a90f-442b-9270-a846bef0d0f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:22.760499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe41c2ca-a90f-442b-9270-a846bef0d0f9 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:22.764124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c934a87e-5659-487e-8585-131800deb4f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:22.764780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c934a87e-5659-487e-8585-131800deb4f6 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:22.765073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c934a87e-5659-487e-8585-131800deb4f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:22.765357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c934a87e-5659-487e-8585-131800deb4f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:22.775069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c934a87e-5659-487e-8585-131800deb4f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:22.776513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c934a87e-5659-487e-8585-131800deb4f6 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:22.785912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1078/2174] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:27.362901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba4a191-48cb-4fe9-8c43-2d016ec56583 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:27.363916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba4a191-48cb-4fe9-8c43-2d016ec56583 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:27.364205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba4a191-48cb-4fe9-8c43-2d016ec56583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:27.393948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-aba4a191-48cb-4fe9-8c43-2d016ec56583 None None] MFA Rules not processed for user `a1e1c24849d7480b8ac804a7e3d2ab5a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:27.395314 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-aba4a191-48cb-4fe9-8c43-2d016ec56583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:27.476297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1097/2175] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:27 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:27.490251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c934a87e-5659-487e-8585-131800deb4f6 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:27.494533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c934a87e-5659-487e-8585-131800deb4f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:27.661494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c934a87e-5659-487e-8585-131800deb4f6 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:27.669779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71970fe8-17d8-4459-b7b5-97e271f8b9e0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:27.670778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71970fe8-17d8-4459-b7b5-97e271f8b9e0 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:27.671224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71970fe8-17d8-4459-b7b5-97e271f8b9e0 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:27.671478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71970fe8-17d8-4459-b7b5-97e271f8b9e0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:27.681052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71970fe8-17d8-4459-b7b5-97e271f8b9e0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:27.682805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71970fe8-17d8-4459-b7b5-97e271f8b9e0 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:27.695372 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1079/2176] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:27 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:29.272033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48901870-888f-4d05-b354-84f80ef67668 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:29.272560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48901870-888f-4d05-b354-84f80ef67668 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:29.272720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48901870-888f-4d05-b354-84f80ef67668 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:29.303339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-48901870-888f-4d05-b354-84f80ef67668 None None] MFA Rules not processed for user `a1e1c24849d7480b8ac804a7e3d2ab5a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:29.304791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-48901870-888f-4d05-b354-84f80ef67668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:29.312331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1098/2177] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:29 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:37.690589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=77,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:39.297866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=40 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:41.488488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71970fe8-17d8-4459-b7b5-97e271f8b9e0 service nova] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:41.490070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-71970fe8-17d8-4459-b7b5-97e271f8b9e0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:41.504249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71970fe8-17d8-4459-b7b5-97e271f8b9e0 service nova] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:41.509014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:41.509643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:41.509925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:41.510203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] PATH_INFO: `/v3/users/e197c5bf77d34c9ba75871d1cd95eab7` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:41.511357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e197c5bf77d34c9ba75871d1cd95eab7)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:41.513308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:41.561972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e197c5bf77d34c9ba75871d1cd95eab7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:41.570055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e197c5bf77d34c9ba75871d1cd95eab7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:41.571005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e197c5bf77d34c9ba75871d1cd95eab7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:41.571661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e197c5bf77d34c9ba75871d1cd95eab7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:41.579777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e197c5bf77d34c9ba75871d1cd95eab7'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:41.597113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1080/2178] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:41 2022] DELETE /identity/v3/users/e197c5bf77d34c9ba75871d1cd95eab7 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:41.601723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b3d0ec-4c79-4bb2-9cdb-efcb8263997f None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:41.602178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b3d0ec-4c79-4bb2-9cdb-efcb8263997f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:41.602356 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b3d0ec-4c79-4bb2-9cdb-efcb8263997f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:41.621948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-14b3d0ec-4c79-4bb2-9cdb-efcb8263997f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:41.623163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-14b3d0ec-4c79-4bb2-9cdb-efcb8263997f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:41.684551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1099/2179] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:41.694914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:41.696328 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:41.754520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcca0c8c-93d0-4ba5-95cd-556682e7f514 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:41.758531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641e9d3c-940d-4b35-a073-b0ae165f4109 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:41.759159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641e9d3c-940d-4b35-a073-b0ae165f4109 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:41.759446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641e9d3c-940d-4b35-a073-b0ae165f4109 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:41.759727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641e9d3c-940d-4b35-a073-b0ae165f4109 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:41.769517 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641e9d3c-940d-4b35-a073-b0ae165f4109 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:41.770961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641e9d3c-940d-4b35-a073-b0ae165f4109 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:41.780845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1081/2180] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:42.406966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b3d0ec-4c79-4bb2-9cdb-efcb8263997f None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:42.408500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-14b3d0ec-4c79-4bb2-9cdb-efcb8263997f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:42.420715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b3d0ec-4c79-4bb2-9cdb-efcb8263997f None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:42.426615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:42.427259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:42.427543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:42.427891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] PATH_INFO: `/v3/projects/908a58724a604e8f8eb8a4c943fe8b7f` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:42.429109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] RBAC: Authorizing `identity:delete_project(project_id=908a58724a604e8f8eb8a4c943fe8b7f)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:42.448025 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:42.490448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '908a58724a604e8f8eb8a4c943fe8b7f', 'request_id': 'req-d3016e30-a5cc-442b-8fae-a32b61491775'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:42.496478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '908a58724a604e8f8eb8a4c943fe8b7f', 'request_id': 'req-d3016e30-a5cc-442b-8fae-a32b61491775'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:42.503710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '908a58724a604e8f8eb8a4c943fe8b7f', 'request_id': 'req-d3016e30-a5cc-442b-8fae-a32b61491775'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:42.514496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] The token cache is being invalidate because project 908a58724a604e8f8eb8a4c943fe8b7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:42.514834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:42.516231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d3016e30-a5cc-442b-8fae-a32b61491775 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:42.517947 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1100/2181] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:42 2022] DELETE /identity/v3/projects/908a58724a604e8f8eb8a4c943fe8b7f => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:42.523214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717b3364-da5f-46e5-bed5-1dbf8bcae6d4 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:42.523745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717b3364-da5f-46e5-bed5-1dbf8bcae6d4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:42.523745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717b3364-da5f-46e5-bed5-1dbf8bcae6d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:42.545749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-717b3364-da5f-46e5-bed5-1dbf8bcae6d4 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:42.547349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-717b3364-da5f-46e5-bed5-1dbf8bcae6d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:42.609793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1082/2182] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:42.617659 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67048a58-c8db-4782-bc3f-22b16e8b1c2c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:42.618121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67048a58-c8db-4782-bc3f-22b16e8b1c2c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:42.618211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67048a58-c8db-4782-bc3f-22b16e8b1c2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:42.637898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-67048a58-c8db-4782-bc3f-22b16e8b1c2c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:42.639216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-67048a58-c8db-4782-bc3f-22b16e8b1c2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:42.648123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1101/2183] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:49:42.652321 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717b3364-da5f-46e5-bed5-1dbf8bcae6d4 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:42.662783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717b3364-da5f-46e5-bed5-1dbf8bcae6d4 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:42.667411 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8952da3d-f446-4a2c-9d25-8f640ed15560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:42.668022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8952da3d-f446-4a2c-9d25-8f640ed15560 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:42.668306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8952da3d-f446-4a2c-9d25-8f640ed15560 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:42.668582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8952da3d-f446-4a2c-9d25-8f640ed15560 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:42.669314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8952da3d-f446-4a2c-9d25-8f640ed15560 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:49:42.669622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8952da3d-f446-4a2c-9d25-8f640ed15560 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:42.671711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8952da3d-f446-4a2c-9d25-8f640ed15560 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:42.683481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1083/2184] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:49:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:49:42.687911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67048a58-c8db-4782-bc3f-22b16e8b1c2c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:42.697059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67048a58-c8db-4782-bc3f-22b16e8b1c2c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:42.701796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629aa160-0eaa-4c2b-a1e0-af96fcc5533e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:42.702403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629aa160-0eaa-4c2b-a1e0-af96fcc5533e admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:42.703231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629aa160-0eaa-4c2b-a1e0-af96fcc5533e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:42.703524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629aa160-0eaa-4c2b-a1e0-af96fcc5533e admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:42.704202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629aa160-0eaa-4c2b-a1e0-af96fcc5533e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:42.706139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629aa160-0eaa-4c2b-a1e0-af96fcc5533e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:42.733330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1102/2185] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:49:42 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:42.737655 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8952da3d-f446-4a2c-9d25-8f640ed15560 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:42.743004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:42.749199 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8952da3d-f446-4a2c-9d25-8f640ed15560 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:42.807203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48bc2c4-8cb3-4049-a0b1-0d4e58f1eb19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:42.807825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48bc2c4-8cb3-4049-a0b1-0d4e58f1eb19 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:42.808120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48bc2c4-8cb3-4049-a0b1-0d4e58f1eb19 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:42.808399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48bc2c4-8cb3-4049-a0b1-0d4e58f1eb19 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:42.809111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48bc2c4-8cb3-4049-a0b1-0d4e58f1eb19 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:42.811036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c48bc2c4-8cb3-4049-a0b1-0d4e58f1eb19 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:42.832449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1084/2186] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:49:42 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:42.836583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629aa160-0eaa-4c2b-a1e0-af96fcc5533e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:42.847452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629aa160-0eaa-4c2b-a1e0-af96fcc5533e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:42.852076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa481785-6a65-4213-8cec-60c8bef6c30b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:42.852713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa481785-6a65-4213-8cec-60c8bef6c30b admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:42.852998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa481785-6a65-4213-8cec-60c8bef6c30b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:42.853275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa481785-6a65-4213-8cec-60c8bef6c30b admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:42.853963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa481785-6a65-4213-8cec-60c8bef6c30b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:42.855547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa481785-6a65-4213-8cec-60c8bef6c30b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:42.865158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1103/2187] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:42 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:42.868846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48bc2c4-8cb3-4049-a0b1-0d4e58f1eb19 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:42.879729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48bc2c4-8cb3-4049-a0b1-0d4e58f1eb19 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:42.884373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ac5037-bc5e-455d-b32f-059e645ed002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:42.885018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ac5037-bc5e-455d-b32f-059e645ed002 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:42.885307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ac5037-bc5e-455d-b32f-059e645ed002 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:42.885586 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ac5037-bc5e-455d-b32f-059e645ed002 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:42.886277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ac5037-bc5e-455d-b32f-059e645ed002 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:42.887902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ac5037-bc5e-455d-b32f-059e645ed002 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:42.898767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1085/2188] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:49:42.902775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa481785-6a65-4213-8cec-60c8bef6c30b admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:42.912595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa481785-6a65-4213-8cec-60c8bef6c30b admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:42.917256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f0ad75-0588-4f9e-b505-c4ab7736fd28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:42.917904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f0ad75-0588-4f9e-b505-c4ab7736fd28 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:42.918190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f0ad75-0588-4f9e-b505-c4ab7736fd28 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:42.918470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f0ad75-0588-4f9e-b505-c4ab7736fd28 admin admin] PATH_INFO: `/v3/projects/fb43a25a59e14dae8ad971f24af58c4b/users/38bedb24ff1940fd8fcc5af30005fd01/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:42.933470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f0ad75-0588-4f9e-b505-c4ab7736fd28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb43a25a59e14dae8ad971f24af58c4b, user_id=38bedb24ff1940fd8fcc5af30005fd01, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:42.936034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f0ad75-0588-4f9e-b505-c4ab7736fd28 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:42.949432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1104/2189] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:49:42 2022] PUT /identity/v3/projects/fb43a25a59e14dae8ad971f24af58c4b/users/38bedb24ff1940fd8fcc5af30005fd01/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:42.960730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6edf88e-0c00-4949-9c59-95b28d0dd0aa None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:42.961019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6edf88e-0c00-4949-9c59-95b28d0dd0aa None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:42.961300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6edf88e-0c00-4949-9c59-95b28d0dd0aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:42.991662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f6edf88e-0c00-4949-9c59-95b28d0dd0aa None None] MFA Rules not processed for user `38bedb24ff1940fd8fcc5af30005fd01`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:42.993014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f6edf88e-0c00-4949-9c59-95b28d0dd0aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:43.059742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1086/2190] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:42 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:49:43.071041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f0ad75-0588-4f9e-b505-c4ab7736fd28 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:43.072782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-31f0ad75-0588-4f9e-b505-c4ab7736fd28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:43.096397 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:43.136083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f0ad75-0588-4f9e-b505-c4ab7736fd28 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:43.140426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaeac62-2502-4733-b0de-58bfc85d2a07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:43.141105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaeac62-2502-4733-b0de-58bfc85d2a07 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:43.141391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaeac62-2502-4733-b0de-58bfc85d2a07 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:43.141671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaeac62-2502-4733-b0de-58bfc85d2a07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:43.150220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afaeac62-2502-4733-b0de-58bfc85d2a07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:43.151644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afaeac62-2502-4733-b0de-58bfc85d2a07 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:43.161032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1105/2191] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:43 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:45.798036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6edf88e-0c00-4949-9c59-95b28d0dd0aa None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:45.860804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6edf88e-0c00-4949-9c59-95b28d0dd0aa None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:45.865484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:45.866126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:45.866410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:45.866685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] PATH_INFO: `/v3/users/38bedb24ff1940fd8fcc5af30005fd01` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:45.867863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] RBAC: Authorizing `identity:delete_user(user_id=38bedb24ff1940fd8fcc5af30005fd01)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:45.869856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:45.883244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:45.916270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38bedb24ff1940fd8fcc5af30005fd01'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:45.924931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38bedb24ff1940fd8fcc5af30005fd01'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:45.925553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38bedb24ff1940fd8fcc5af30005fd01'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:45.926167 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38bedb24ff1940fd8fcc5af30005fd01'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:45.933346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38bedb24ff1940fd8fcc5af30005fd01'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:45.953669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1087/2192] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:49:45 2022] DELETE /identity/v3/users/38bedb24ff1940fd8fcc5af30005fd01 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:45.958257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ebc8a-c251-4a3a-b44f-204ab99651e5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:45.958748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ebc8a-c251-4a3a-b44f-204ab99651e5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:45.959013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ebc8a-c251-4a3a-b44f-204ab99651e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:45.978865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-082ebc8a-c251-4a3a-b44f-204ab99651e5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:45.980405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-082ebc8a-c251-4a3a-b44f-204ab99651e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:46.039975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1106/2193] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:46.050144 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:46.051587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:46.108235 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd048c1-09a1-401f-8a22-4fcf1f2bc11d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:46.111957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315bacc8-07a2-4258-bc52-b9f9cca5e1c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:46.112512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315bacc8-07a2-4258-bc52-b9f9cca5e1c4 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.112827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315bacc8-07a2-4258-bc52-b9f9cca5e1c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.113107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315bacc8-07a2-4258-bc52-b9f9cca5e1c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:46.122093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315bacc8-07a2-4258-bc52-b9f9cca5e1c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:46.123523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315bacc8-07a2-4258-bc52-b9f9cca5e1c4 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:46.133387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1088/2194] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:46.413577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082ebc8a-c251-4a3a-b44f-204ab99651e5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:46.415358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-082ebc8a-c251-4a3a-b44f-204ab99651e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:46.428297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082ebc8a-c251-4a3a-b44f-204ab99651e5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:46.432909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:46.433557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.433841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.434118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] PATH_INFO: `/v3/projects/fb43a25a59e14dae8ad971f24af58c4b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:46.436487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb43a25a59e14dae8ad971f24af58c4b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:46.438549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:46.487834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb43a25a59e14dae8ad971f24af58c4b', 'request_id': 'req-cfe4e608-3c81-4859-9446-5e44364573d3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:46.494047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb43a25a59e14dae8ad971f24af58c4b', 'request_id': 'req-cfe4e608-3c81-4859-9446-5e44364573d3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:46.500821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb43a25a59e14dae8ad971f24af58c4b', 'request_id': 'req-cfe4e608-3c81-4859-9446-5e44364573d3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:46.511148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] The token cache is being invalidate because project fb43a25a59e14dae8ad971f24af58c4b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:49:46.511446 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:46.512079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cfe4e608-3c81-4859-9446-5e44364573d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:49:46.513622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1107/2195] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:49:46 2022] DELETE /identity/v3/projects/fb43a25a59e14dae8ad971f24af58c4b => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:46.519281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7072f6c3-0515-4332-a302-980b0b0d012f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.519573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7072f6c3-0515-4332-a302-980b0b0d012f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.519855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7072f6c3-0515-4332-a302-980b0b0d012f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:46.541685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-7072f6c3-0515-4332-a302-980b0b0d012f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:46.543185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7072f6c3-0515-4332-a302-980b0b0d012f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:46.612389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1089/2196] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:46.620006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ce6894-c14d-4e22-9ade-461a0092f82c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.620303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ce6894-c14d-4e22-9ade-461a0092f82c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.620589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ce6894-c14d-4e22-9ade-461a0092f82c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:46.642570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a9ce6894-c14d-4e22-9ade-461a0092f82c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:46.643818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a9ce6894-c14d-4e22-9ade-461a0092f82c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:46.658060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1108/2197] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:46.662265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7072f6c3-0515-4332-a302-980b0b0d012f None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:46.672864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7072f6c3-0515-4332-a302-980b0b0d012f None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:46.678210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c0253-3500-4d3f-b5fd-75a740e22f46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:46.678882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c0253-3500-4d3f-b5fd-75a740e22f46 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.679170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c0253-3500-4d3f-b5fd-75a740e22f46 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.679480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18c0253-3500-4d3f-b5fd-75a740e22f46 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:46.680186 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18c0253-3500-4d3f-b5fd-75a740e22f46 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:49:46.680494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18c0253-3500-4d3f-b5fd-75a740e22f46 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:46.682218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18c0253-3500-4d3f-b5fd-75a740e22f46 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:46.698483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1090/2198] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:49:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:49:46.700160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ce6894-c14d-4e22-9ade-461a0092f82c None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:46.712338 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ce6894-c14d-4e22-9ade-461a0092f82c None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:46.717204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee8e91c-0305-43c4-9d43-7c6e51941429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:46.717879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee8e91c-0305-43c4-9d43-7c6e51941429 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.718203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee8e91c-0305-43c4-9d43-7c6e51941429 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.718485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee8e91c-0305-43c4-9d43-7c6e51941429 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:46.719171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee8e91c-0305-43c4-9d43-7c6e51941429 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:46.721194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee8e91c-0305-43c4-9d43-7c6e51941429 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:46.748709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1109/2199] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:49:46 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:49:46.753544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c0253-3500-4d3f-b5fd-75a740e22f46 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:46.763537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18c0253-3500-4d3f-b5fd-75a740e22f46 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:46.782035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:46.825835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538a3cf2-8345-4c9e-9d49-1e0eb543e364 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:46.826518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538a3cf2-8345-4c9e-9d49-1e0eb543e364 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.826749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538a3cf2-8345-4c9e-9d49-1e0eb543e364 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.827017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538a3cf2-8345-4c9e-9d49-1e0eb543e364 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:46.828256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538a3cf2-8345-4c9e-9d49-1e0eb543e364 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:46.830314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538a3cf2-8345-4c9e-9d49-1e0eb543e364 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:46.852905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1091/2200] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:49:46 2022] POST /identity/v3/users => generated 287 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:49:46.856804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee8e91c-0305-43c4-9d43-7c6e51941429 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:46.869415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee8e91c-0305-43c4-9d43-7c6e51941429 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:46.876722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3995a1f-0e9a-4ac1-80c8-cbf450d5bcb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:46.877422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3995a1f-0e9a-4ac1-80c8-cbf450d5bcb4 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.877745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3995a1f-0e9a-4ac1-80c8-cbf450d5bcb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.878179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3995a1f-0e9a-4ac1-80c8-cbf450d5bcb4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:46.879156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3995a1f-0e9a-4ac1-80c8-cbf450d5bcb4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:46.881297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3995a1f-0e9a-4ac1-80c8-cbf450d5bcb4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:46.894983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1110/2201] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:46 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:46.898987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538a3cf2-8345-4c9e-9d49-1e0eb543e364 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:46.909929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538a3cf2-8345-4c9e-9d49-1e0eb543e364 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:46.914783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d7f7ec-da93-4a78-aaca-ca393f3818c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:46.915325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d7f7ec-da93-4a78-aaca-ca393f3818c6 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.915602 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d7f7ec-da93-4a78-aaca-ca393f3818c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.915913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d7f7ec-da93-4a78-aaca-ca393f3818c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:46.916583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d7f7ec-da93-4a78-aaca-ca393f3818c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:46.918202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d7f7ec-da93-4a78-aaca-ca393f3818c6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:46.930182 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1092/2202] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:49:46 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:49:46.934102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3995a1f-0e9a-4ac1-80c8-cbf450d5bcb4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:46.944681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3995a1f-0e9a-4ac1-80c8-cbf450d5bcb4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:46.949303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7cb018-68bb-4009-b506-b830391d5435 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:46.949962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7cb018-68bb-4009-b506-b830391d5435 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.950250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7cb018-68bb-4009-b506-b830391d5435 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.950528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7cb018-68bb-4009-b506-b830391d5435 admin admin] PATH_INFO: `/v3/projects/48e14495c8f7484ab3adc511e812d3e8/users/e758990a520b45a0b5c294972f0dd37a/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:46.953385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:49:46.970178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7cb018-68bb-4009-b506-b830391d5435 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48e14495c8f7484ab3adc511e812d3e8, user_id=e758990a520b45a0b5c294972f0dd37a, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:46.972883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7cb018-68bb-4009-b506-b830391d5435 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:46.985952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1111/2203] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:49:46 2022] PUT /identity/v3/projects/48e14495c8f7484ab3adc511e812d3e8/users/e758990a520b45a0b5c294972f0dd37a/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:49:46.996879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd248088-9d34-4208-a121-c37fa9857cbf None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:46.997168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd248088-9d34-4208-a121-c37fa9857cbf None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:46.997445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd248088-9d34-4208-a121-c37fa9857cbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:47.027113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-cd248088-9d34-4208-a121-c37fa9857cbf None None] MFA Rules not processed for user `e758990a520b45a0b5c294972f0dd37a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:47.028574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-cd248088-9d34-4208-a121-c37fa9857cbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:47.099672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1093/2204] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:46 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:47.110910 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7cb018-68bb-4009-b506-b830391d5435 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:47.112620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2e7cb018-68bb-4009-b506-b830391d5435 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:47.167466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7cb018-68bb-4009-b506-b830391d5435 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:47.171908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67da7f53-636c-4804-aa2e-4715dcd721a9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:47.172595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67da7f53-636c-4804-aa2e-4715dcd721a9 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:47.172912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67da7f53-636c-4804-aa2e-4715dcd721a9 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:47.173189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67da7f53-636c-4804-aa2e-4715dcd721a9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:47.181580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67da7f53-636c-4804-aa2e-4715dcd721a9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:47.182998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67da7f53-636c-4804-aa2e-4715dcd721a9 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:47.192517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1112/2205] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:49:47 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:49:52.765177 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72c0aa3-9957-4b03-abdb-741c2d41296e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:52.765883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72c0aa3-9957-4b03-abdb-741c2d41296e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:52.766277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72c0aa3-9957-4b03-abdb-741c2d41296e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:52.767084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e296fe-cf6f-45fd-af37-6dfb50eef350 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:52.767626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e296fe-cf6f-45fd-af37-6dfb50eef350 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:52.767920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e296fe-cf6f-45fd-af37-6dfb50eef350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:52.789231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f72c0aa3-9957-4b03-abdb-741c2d41296e None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:52.808684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f72c0aa3-9957-4b03-abdb-741c2d41296e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:52.808804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-e3e296fe-cf6f-45fd-af37-6dfb50eef350 None None] MFA Rules not processed for user `e758990a520b45a0b5c294972f0dd37a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:49:52.808804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e3e296fe-cf6f-45fd-af37-6dfb50eef350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:52.808804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1113/2206] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:49:52 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:52.860749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1094/2207] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:49:52 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:49:52.900254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e296fe-cf6f-45fd-af37-6dfb50eef350 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:49:52.901708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e3e296fe-cf6f-45fd-af37-6dfb50eef350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:49:52.975029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e296fe-cf6f-45fd-af37-6dfb50eef350 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:49:52.979941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabf0884-f313-426c-a1ee-d9f01c82f6ae service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:49:52.980557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf0884-f313-426c-a1ee-d9f01c82f6ae service swift] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:49:52.980871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf0884-f313-426c-a1ee-d9f01c82f6ae service swift] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:49:52.981155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf0884-f313-426c-a1ee-d9f01c82f6ae service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:49:52.989508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabf0884-f313-426c-a1ee-d9f01c82f6ae service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:49:52.991128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabf0884-f313-426c-a1ee-d9f01c82f6ae service swift] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:49:52.999317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1114/2208] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:49:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:50:02.853413 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=82,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:02.995879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=68,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:13.632730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0973cc90-4e2f-4d03-a054-02bf47877788 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:13.633244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0973cc90-4e2f-4d03-a054-02bf47877788 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:13.633341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0973cc90-4e2f-4d03-a054-02bf47877788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:13.654498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0973cc90-4e2f-4d03-a054-02bf47877788 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:13.656293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0973cc90-4e2f-4d03-a054-02bf47877788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:13.663532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1095/2209] 10.222.0.40 () {58 vars in 967 bytes} [Fri Jul 29 04:50:13 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:50:13.676215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabf0884-f313-426c-a1ee-d9f01c82f6ae service swift] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:13.687708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabf0884-f313-426c-a1ee-d9f01c82f6ae service swift] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:13.693221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c91fd4-ca87-4d37-9cea-db6759501e26 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:13.693842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c91fd4-ca87-4d37-9cea-db6759501e26 service swift] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:13.694133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c91fd4-ca87-4d37-9cea-db6759501e26 service swift] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:13.694415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c91fd4-ca87-4d37-9cea-db6759501e26 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:13.703539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c91fd4-ca87-4d37-9cea-db6759501e26 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:13.705175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c91fd4-ca87-4d37-9cea-db6759501e26 service swift] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:13.713607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1115/2210] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:50:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:50:15.699908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0973cc90-4e2f-4d03-a054-02bf47877788 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:15.701425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0973cc90-4e2f-4d03-a054-02bf47877788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:15.780921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0973cc90-4e2f-4d03-a054-02bf47877788 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:15.787029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:15.787777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:15.788315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:15.788647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] PATH_INFO: `/v3/users/87f97b8b406e442b93cbdda090d7e9ff` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:15.790016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=87f97b8b406e442b93cbdda090d7e9ff)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:15.791964 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:15.856305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '87f97b8b406e442b93cbdda090d7e9ff'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:15.867347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '87f97b8b406e442b93cbdda090d7e9ff'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:15.867974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '87f97b8b406e442b93cbdda090d7e9ff'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:15.868582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '87f97b8b406e442b93cbdda090d7e9ff'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:15.878171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d55f24d-3c5b-4281-bf79-372adf4159c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '87f97b8b406e442b93cbdda090d7e9ff'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:15.899275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1096/2211] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:50:15 2022] DELETE /identity/v3/users/87f97b8b406e442b93cbdda090d7e9ff => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:16.094452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c91fd4-ca87-4d37-9cea-db6759501e26 service swift] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:16.105919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c91fd4-ca87-4d37-9cea-db6759501e26 service swift] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:16.181210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:16.181864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.182149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.182429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] PATH_INFO: `/v3/projects/e52663d55be44e12b4aa939974e8e517` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.183607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e52663d55be44e12b4aa939974e8e517)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:16.185700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:16.232997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e52663d55be44e12b4aa939974e8e517', 'request_id': 'req-09cef004-6744-48f7-b10f-91896d365c7d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:16.239040 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e52663d55be44e12b4aa939974e8e517', 'request_id': 'req-09cef004-6744-48f7-b10f-91896d365c7d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:16.246463 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e52663d55be44e12b4aa939974e8e517', 'request_id': 'req-09cef004-6744-48f7-b10f-91896d365c7d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:16.259014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] The token cache is being invalidate because project e52663d55be44e12b4aa939974e8e517 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:50:16.259322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:16.259950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-09cef004-6744-48f7-b10f-91896d365c7d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:16.261499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1116/2212] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:50:16 2022] DELETE /identity/v3/projects/e52663d55be44e12b4aa939974e8e517 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:16.267064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f92d195-f71b-4b32-8c52-027bdef58bf7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.267298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f92d195-f71b-4b32-8c52-027bdef58bf7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.267583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f92d195-f71b-4b32-8c52-027bdef58bf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.287553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6f92d195-f71b-4b32-8c52-027bdef58bf7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:16.288804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6f92d195-f71b-4b32-8c52-027bdef58bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:16.349332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1097/2213] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:16.356952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f4985a-8c2f-4227-9f11-f1a183211dd3 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.357286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f4985a-8c2f-4227-9f11-f1a183211dd3 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.357637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f4985a-8c2f-4227-9f11-f1a183211dd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.377166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-66f4985a-8c2f-4227-9f11-f1a183211dd3 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:16.378377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-66f4985a-8c2f-4227-9f11-f1a183211dd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:16.386572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1117/2214] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:50:16.391585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f92d195-f71b-4b32-8c52-027bdef58bf7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:16.401248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f92d195-f71b-4b32-8c52-027bdef58bf7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:16.406722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2f0196-cef0-4e7d-94b7-c68f87e6564e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:16.407341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2f0196-cef0-4e7d-94b7-c68f87e6564e admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.407622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2f0196-cef0-4e7d-94b7-c68f87e6564e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.407896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2f0196-cef0-4e7d-94b7-c68f87e6564e admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.408592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2f0196-cef0-4e7d-94b7-c68f87e6564e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:50:16.408923 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2f0196-cef0-4e7d-94b7-c68f87e6564e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:16.410524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2f0196-cef0-4e7d-94b7-c68f87e6564e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:16.419064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:16.423359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1098/2215] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:50:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:50:16.428146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f4985a-8c2f-4227-9f11-f1a183211dd3 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:16.437905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f4985a-8c2f-4227-9f11-f1a183211dd3 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:16.442477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b183c1e-b81d-42c2-96f8-87a8848487a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:16.443081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b183c1e-b81d-42c2-96f8-87a8848487a2 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.443377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b183c1e-b81d-42c2-96f8-87a8848487a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.443643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b183c1e-b81d-42c2-96f8-87a8848487a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.444298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b183c1e-b81d-42c2-96f8-87a8848487a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:16.446226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b183c1e-b81d-42c2-96f8-87a8848487a2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:16.474119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1118/2216] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:50:16 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:50:16.478310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2f0196-cef0-4e7d-94b7-c68f87e6564e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:16.488230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2f0196-cef0-4e7d-94b7-c68f87e6564e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:16.547035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aad20e-d60f-4985-879e-77a87623d3c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:16.547573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aad20e-d60f-4985-879e-77a87623d3c4 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.547854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aad20e-d60f-4985-879e-77a87623d3c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.548131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aad20e-d60f-4985-879e-77a87623d3c4 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.548845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aad20e-d60f-4985-879e-77a87623d3c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:16.550782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aad20e-d60f-4985-879e-77a87623d3c4 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:16.572564 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1099/2217] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:50:16 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:50:16.577387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b183c1e-b81d-42c2-96f8-87a8848487a2 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:16.588045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b183c1e-b81d-42c2-96f8-87a8848487a2 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:16.593655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f473b5-156a-438d-85cc-1cfa855c5a49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:16.594277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f473b5-156a-438d-85cc-1cfa855c5a49 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.594565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f473b5-156a-438d-85cc-1cfa855c5a49 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.594845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f473b5-156a-438d-85cc-1cfa855c5a49 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.595555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f473b5-156a-438d-85cc-1cfa855c5a49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:16.597190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f473b5-156a-438d-85cc-1cfa855c5a49 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:16.607564 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1119/2218] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:50:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:50:16.611910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aad20e-d60f-4985-879e-77a87623d3c4 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:16.623560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aad20e-d60f-4985-879e-77a87623d3c4 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:16.628523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c825813-273a-46f0-8b36-2b13b2fc3388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:16.629279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c825813-273a-46f0-8b36-2b13b2fc3388 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.629633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c825813-273a-46f0-8b36-2b13b2fc3388 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.629981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c825813-273a-46f0-8b36-2b13b2fc3388 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.631089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c825813-273a-46f0-8b36-2b13b2fc3388 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:16.632921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c825813-273a-46f0-8b36-2b13b2fc3388 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:16.645424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1100/2219] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:50:16 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:50:16.649364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f473b5-156a-438d-85cc-1cfa855c5a49 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:16.660789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f473b5-156a-438d-85cc-1cfa855c5a49 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:16.665722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5223743a-67ba-4294-bf0f-a5e7d54e5df8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:16.666583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5223743a-67ba-4294-bf0f-a5e7d54e5df8 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.666950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5223743a-67ba-4294-bf0f-a5e7d54e5df8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.667415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5223743a-67ba-4294-bf0f-a5e7d54e5df8 admin admin] PATH_INFO: `/v3/projects/5f6ad2154b3c4745be93fcae2a0181d1/users/b71589e975a443cbb4726423d547507b/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.683143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5223743a-67ba-4294-bf0f-a5e7d54e5df8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f6ad2154b3c4745be93fcae2a0181d1, user_id=b71589e975a443cbb4726423d547507b, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:16.686795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5223743a-67ba-4294-bf0f-a5e7d54e5df8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:16.701329 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1120/2220] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:50:16 2022] PUT /identity/v3/projects/5f6ad2154b3c4745be93fcae2a0181d1/users/b71589e975a443cbb4726423d547507b/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:50:16.712119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc75777-2d1e-4d01-9391-b80d609c4a06 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.713110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc75777-2d1e-4d01-9391-b80d609c4a06 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.713110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc75777-2d1e-4d01-9391-b80d609c4a06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.741677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-2cc75777-2d1e-4d01-9391-b80d609c4a06 None None] MFA Rules not processed for user `b71589e975a443cbb4726423d547507b`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:16.742944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2cc75777-2d1e-4d01-9391-b80d609c4a06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:16.812513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1101/2221] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:16 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:16.823441 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5223743a-67ba-4294-bf0f-a5e7d54e5df8 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:16.825246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5223743a-67ba-4294-bf0f-a5e7d54e5df8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:16.839813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:16.901429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5223743a-67ba-4294-bf0f-a5e7d54e5df8 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:16.905719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca279fcb-dcd3-4dc9-bbc0-80984aba2301 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:16.906957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca279fcb-dcd3-4dc9-bbc0-80984aba2301 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:16.907244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca279fcb-dcd3-4dc9-bbc0-80984aba2301 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:16.907524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca279fcb-dcd3-4dc9-bbc0-80984aba2301 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:16.917009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca279fcb-dcd3-4dc9-bbc0-80984aba2301 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:16.918484 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca279fcb-dcd3-4dc9-bbc0-80984aba2301 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:16.928389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1121/2222] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:50:16 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:50:24.079446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc75777-2d1e-4d01-9391-b80d609c4a06 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:24.080903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2cc75777-2d1e-4d01-9391-b80d609c4a06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:24.144776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc75777-2d1e-4d01-9391-b80d609c4a06 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:24.149647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:24.150294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:24.150587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:24.150884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] PATH_INFO: `/v3/users/3bc686e22f604f5ebb144cd0c1c7748c` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:24.152062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bc686e22f604f5ebb144cd0c1c7748c)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:24.154053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:24.165323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:24.204455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bc686e22f604f5ebb144cd0c1c7748c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:24.212261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bc686e22f604f5ebb144cd0c1c7748c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:24.213098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bc686e22f604f5ebb144cd0c1c7748c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:24.213773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bc686e22f604f5ebb144cd0c1c7748c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:24.223484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bc686e22f604f5ebb144cd0c1c7748c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:24.242735 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1102/2223] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:50:24 2022] DELETE /identity/v3/users/3bc686e22f604f5ebb144cd0c1c7748c => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:24.247077 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f395d2f-e1d1-49c0-b5ec-d633055f3659 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:24.247532 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f395d2f-e1d1-49c0-b5ec-d633055f3659 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:24.247626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f395d2f-e1d1-49c0-b5ec-d633055f3659 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:24.267722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-9f395d2f-e1d1-49c0-b5ec-d633055f3659 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:24.268801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9f395d2f-e1d1-49c0-b5ec-d633055f3659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:24.336429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1122/2224] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:24.346676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:24.348629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:24.415755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7393a1-bb15-4626-adb4-c8a37fe1c1f4 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:24.419721 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c7366e-b86c-47d0-8db2-d8a239dc4a60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:24.420414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c7366e-b86c-47d0-8db2-d8a239dc4a60 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:24.421248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c7366e-b86c-47d0-8db2-d8a239dc4a60 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:24.421525 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c7366e-b86c-47d0-8db2-d8a239dc4a60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:24.433620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c7366e-b86c-47d0-8db2-d8a239dc4a60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:24.435075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c7366e-b86c-47d0-8db2-d8a239dc4a60 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:24.446109 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1103/2225] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:50:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:50:24.739932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f395d2f-e1d1-49c0-b5ec-d633055f3659 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:24.742130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9f395d2f-e1d1-49c0-b5ec-d633055f3659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:24.759611 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f395d2f-e1d1-49c0-b5ec-d633055f3659 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:24.770497 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:24.771382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:24.771686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:24.771971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] PATH_INFO: `/v3/projects/fb2577b0b5a946a5beee64c611d5b768` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:24.773686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb2577b0b5a946a5beee64c611d5b768)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:24.775953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:24.858924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb2577b0b5a946a5beee64c611d5b768', 'request_id': 'req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:24.865712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb2577b0b5a946a5beee64c611d5b768', 'request_id': 'req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:24.875166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb2577b0b5a946a5beee64c611d5b768', 'request_id': 'req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:24.890286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] The token cache is being invalidate because project fb2577b0b5a946a5beee64c611d5b768 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:50:24.890585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:24.891516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-e2718655-1a2e-4bfe-b4dd-54f1fa5503ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:24.893204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1123/2226] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:50:24 2022] DELETE /identity/v3/projects/fb2577b0b5a946a5beee64c611d5b768 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:25.196861 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c7366e-b86c-47d0-8db2-d8a239dc4a60 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:25.196861 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-02c7366e-b86c-47d0-8db2-d8a239dc4a60 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:25.267504 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c7366e-b86c-47d0-8db2-d8a239dc4a60 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:25.307491 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:25.344788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb0167c-10bf-4b4d-ab2c-c89537ea4175 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:25.345637 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb0167c-10bf-4b4d-ab2c-c89537ea4175 service placement] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:25.345934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb0167c-10bf-4b4d-ab2c-c89537ea4175 service placement] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:25.346289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb0167c-10bf-4b4d-ab2c-c89537ea4175 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:25.348159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6eb0167c-10bf-4b4d-ab2c-c89537ea4175 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:25.360509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb0167c-10bf-4b4d-ab2c-c89537ea4175 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:25.362622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb0167c-10bf-4b4d-ab2c-c89537ea4175 service placement] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:25.382830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1104/2227] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:50:25 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:50:29.003316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcad8a15-2e02-4b71-a36d-c1001108f064 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:29.003759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcad8a15-2e02-4b71-a36d-c1001108f064 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:29.003879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcad8a15-2e02-4b71-a36d-c1001108f064 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:29.024092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-dcad8a15-2e02-4b71-a36d-c1001108f064 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:29.025376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-dcad8a15-2e02-4b71-a36d-c1001108f064 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:29.059281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:29.082328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1124/2228] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:50:29 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:29.095248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb0167c-10bf-4b4d-ab2c-c89537ea4175 service placement] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:29.096670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6eb0167c-10bf-4b4d-ab2c-c89537ea4175 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:29.163201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb0167c-10bf-4b4d-ab2c-c89537ea4175 service placement] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:29.168183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df373ee-612d-4128-ab38-908f65b4ee37 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:29.168821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df373ee-612d-4128-ab38-908f65b4ee37 service swift] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:29.169365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df373ee-612d-4128-ab38-908f65b4ee37 service swift] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:29.169645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df373ee-612d-4128-ab38-908f65b4ee37 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:29.177960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df373ee-612d-4128-ab38-908f65b4ee37 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:29.179490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df373ee-612d-4128-ab38-908f65b4ee37 service swift] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:29.187652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1105/2229] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:50:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:50:29.714387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5648c462-e729-46d7-bc29-ef814636cee5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:29.714681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5648c462-e729-46d7-bc29-ef814636cee5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:29.714959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5648c462-e729-46d7-bc29-ef814636cee5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:29.734648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-5648c462-e729-46d7-bc29-ef814636cee5 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:29.735860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5648c462-e729-46d7-bc29-ef814636cee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:29.742937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1125/2230] 10.222.0.40 () {58 vars in 967 bytes} [Fri Jul 29 04:50:29 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:29.756073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df373ee-612d-4128-ab38-908f65b4ee37 service swift] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:29.766790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df373ee-612d-4128-ab38-908f65b4ee37 service swift] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:29.771993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c992b1-f85e-4adc-af4e-024eb903c90b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:29.772627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c992b1-f85e-4adc-af4e-024eb903c90b service swift] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:29.772948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c992b1-f85e-4adc-af4e-024eb903c90b service swift] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:29.773231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c992b1-f85e-4adc-af4e-024eb903c90b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:29.781490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c992b1-f85e-4adc-af4e-024eb903c90b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:29.783085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c992b1-f85e-4adc-af4e-024eb903c90b service swift] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:29.791242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1106/2231] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:50:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:50:30.026081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3235ec7-8de1-4328-b0da-d670a5206144 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:30.026547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3235ec7-8de1-4328-b0da-d670a5206144 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:30.026637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3235ec7-8de1-4328-b0da-d670a5206144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:30.046994 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a3235ec7-8de1-4328-b0da-d670a5206144 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:30.048189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a3235ec7-8de1-4328-b0da-d670a5206144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:30.055695 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1126/2232] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:50:30 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:30.068079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c992b1-f85e-4adc-af4e-024eb903c90b service swift] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:30.079429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c992b1-f85e-4adc-af4e-024eb903c90b service swift] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:30.084466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453950d5-a4a2-432e-9e5f-53c928de0a2f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:30.085112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453950d5-a4a2-432e-9e5f-53c928de0a2f service swift] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:30.085403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453950d5-a4a2-432e-9e5f-53c928de0a2f service swift] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:30.085686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453950d5-a4a2-432e-9e5f-53c928de0a2f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:30.094896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453950d5-a4a2-432e-9e5f-53c928de0a2f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:30.096516 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453950d5-a4a2-432e-9e5f-53c928de0a2f service swift] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:30.104709 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1107/2233] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:50:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:50:30.363072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3235ec7-8de1-4328-b0da-d670a5206144 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:30.364507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a3235ec7-8de1-4328-b0da-d670a5206144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:30.430635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e29a21f-0771-4a6b-ad61-3c95f5e99d1f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:30.430927 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e29a21f-0771-4a6b-ad61-3c95f5e99d1f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:30.431237 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e29a21f-0771-4a6b-ad61-3c95f5e99d1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:30.435021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3235ec7-8de1-4328-b0da-d670a5206144 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:30.441172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:30.442017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:30.442866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:30.443164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] PATH_INFO: `/v3/users/b71589e975a443cbb4726423d547507b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:30.444381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b71589e975a443cbb4726423d547507b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:30.446811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:30.450695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-7e29a21f-0771-4a6b-ad61-3c95f5e99d1f None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:30.452771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7e29a21f-0771-4a6b-ad61-3c95f5e99d1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:30.459811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1108/2234] 10.222.0.40 () {58 vars in 967 bytes} [Fri Jul 29 04:50:30 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:50:30.472453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e29a21f-0771-4a6b-ad61-3c95f5e99d1f None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:30.485694 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e29a21f-0771-4a6b-ad61-3c95f5e99d1f None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:30.491386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233cdde5-51d1-43cd-9946-f1fa0b3ad1fb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:30.492009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233cdde5-51d1-43cd-9946-f1fa0b3ad1fb service swift] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:30.492302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233cdde5-51d1-43cd-9946-f1fa0b3ad1fb service swift] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:30.492728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233cdde5-51d1-43cd-9946-f1fa0b3ad1fb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:30.497654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b71589e975a443cbb4726423d547507b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:30.498315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b71589e975a443cbb4726423d547507b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:30.499561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b71589e975a443cbb4726423d547507b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:30.501114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233cdde5-51d1-43cd-9946-f1fa0b3ad1fb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:30.502730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233cdde5-51d1-43cd-9946-f1fa0b3ad1fb service swift] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:30.511120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1109/2235] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:50:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 29 04:50:30.511890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b71589e975a443cbb4726423d547507b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:30.519734 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b71589e975a443cbb4726423d547507b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:30.535401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1127/2236] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:50:30 2022] DELETE /identity/v3/users/b71589e975a443cbb4726423d547507b => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:30.539745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5b588-8881-46d2-958f-2a18a9da1043 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:30.540252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5b588-8881-46d2-958f-2a18a9da1043 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:30.540536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5b588-8881-46d2-958f-2a18a9da1043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:30.560827 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-fae5b588-8881-46d2-958f-2a18a9da1043 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:30.562193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fae5b588-8881-46d2-958f-2a18a9da1043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:30.629825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1110/2237] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:30.638215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:30.640478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:30.709577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d1129d-220e-4526-a311-ad8c6dacc178 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:30.716773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719f6da2-031f-4312-9f8a-79a04623f0c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:30.716773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719f6da2-031f-4312-9f8a-79a04623f0c3 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:30.716773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719f6da2-031f-4312-9f8a-79a04623f0c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:30.716773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719f6da2-031f-4312-9f8a-79a04623f0c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:30.722392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:30.726552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719f6da2-031f-4312-9f8a-79a04623f0c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:30.729573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719f6da2-031f-4312-9f8a-79a04623f0c3 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:30.745461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1128/2238] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:50:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:50:31.034928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae5b588-8881-46d2-958f-2a18a9da1043 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:31.036255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fae5b588-8881-46d2-958f-2a18a9da1043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:31.064404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae5b588-8881-46d2-958f-2a18a9da1043 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:31.077167 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:31.077755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.078025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.078308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] PATH_INFO: `/v3/projects/5f6ad2154b3c4745be93fcae2a0181d1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.079507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f6ad2154b3c4745be93fcae2a0181d1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:31.081563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:31.083394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:31.137462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f6ad2154b3c4745be93fcae2a0181d1', 'request_id': 'req-71aa25a0-1580-4f78-aec0-e04d34d660f6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:31.148447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f6ad2154b3c4745be93fcae2a0181d1', 'request_id': 'req-71aa25a0-1580-4f78-aec0-e04d34d660f6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:31.156641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f6ad2154b3c4745be93fcae2a0181d1', 'request_id': 'req-71aa25a0-1580-4f78-aec0-e04d34d660f6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:31.171600 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] The token cache is being invalidate because project 5f6ad2154b3c4745be93fcae2a0181d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:50:31.171943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:31.172671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71aa25a0-1580-4f78-aec0-e04d34d660f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:31.174432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1111/2239] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:50:31 2022] DELETE /identity/v3/projects/5f6ad2154b3c4745be93fcae2a0181d1 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:31.179723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9075f8a-121e-42ac-acaa-4d584a93fcae None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.180388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9075f8a-121e-42ac-acaa-4d584a93fcae None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.180388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9075f8a-121e-42ac-acaa-4d584a93fcae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.211198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f9075f8a-121e-42ac-acaa-4d584a93fcae None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:31.212572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f9075f8a-121e-42ac-acaa-4d584a93fcae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:31.279323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1129/2240] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:31.289706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38088e18-be70-4b91-8f65-fc551e8d2f6d None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.290235 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38088e18-be70-4b91-8f65-fc551e8d2f6d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.290523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38088e18-be70-4b91-8f65-fc551e8d2f6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.314954 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-38088e18-be70-4b91-8f65-fc551e8d2f6d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:31.316191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-38088e18-be70-4b91-8f65-fc551e8d2f6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:31.348817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1112/2241] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:31.353189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9075f8a-121e-42ac-acaa-4d584a93fcae None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:31.366015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9075f8a-121e-42ac-acaa-4d584a93fcae None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:31.370933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e920a69-ea1c-4752-8f21-37bc1aef018d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:31.371555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e920a69-ea1c-4752-8f21-37bc1aef018d admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.371842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e920a69-ea1c-4752-8f21-37bc1aef018d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.372203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e920a69-ea1c-4752-8f21-37bc1aef018d admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.372969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e920a69-ea1c-4752-8f21-37bc1aef018d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:50:31.373285 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e920a69-ea1c-4752-8f21-37bc1aef018d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:31.375461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e920a69-ea1c-4752-8f21-37bc1aef018d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:31.387613 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1130/2242] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:50:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:50:31.392193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38088e18-be70-4b91-8f65-fc551e8d2f6d None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:31.401759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38088e18-be70-4b91-8f65-fc551e8d2f6d None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:31.406604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77940d85-6a07-4800-b0cb-fdbf50269767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:31.407221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77940d85-6a07-4800-b0cb-fdbf50269767 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.407514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77940d85-6a07-4800-b0cb-fdbf50269767 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.407786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77940d85-6a07-4800-b0cb-fdbf50269767 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.408766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77940d85-6a07-4800-b0cb-fdbf50269767 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:31.410668 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77940d85-6a07-4800-b0cb-fdbf50269767 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:31.440823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1113/2243] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:50:31 2022] POST /identity/v3/projects => generated 397 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:50:31.444367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e920a69-ea1c-4752-8f21-37bc1aef018d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:31.454795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e920a69-ea1c-4752-8f21-37bc1aef018d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:31.513476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d76e9ce-07d0-45cb-9a27-807e13c5ecc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:31.514256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d76e9ce-07d0-45cb-9a27-807e13c5ecc4 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.514543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d76e9ce-07d0-45cb-9a27-807e13c5ecc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.514829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d76e9ce-07d0-45cb-9a27-807e13c5ecc4 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.515506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d76e9ce-07d0-45cb-9a27-807e13c5ecc4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:31.517518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d76e9ce-07d0-45cb-9a27-807e13c5ecc4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:31.545993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1131/2244] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:50:31 2022] POST /identity/v3/users => generated 299 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:50:31.547337 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77940d85-6a07-4800-b0cb-fdbf50269767 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:31.558669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77940d85-6a07-4800-b0cb-fdbf50269767 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:31.564686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e141c6-1ce8-4259-b8ae-7830e8f93967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:31.565621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e141c6-1ce8-4259-b8ae-7830e8f93967 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.566112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e141c6-1ce8-4259-b8ae-7830e8f93967 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.566787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e141c6-1ce8-4259-b8ae-7830e8f93967 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.567593 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e141c6-1ce8-4259-b8ae-7830e8f93967 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:31.569676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e141c6-1ce8-4259-b8ae-7830e8f93967 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:31.582823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1114/2245] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:50:31 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:50:31.587087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d76e9ce-07d0-45cb-9a27-807e13c5ecc4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:31.598452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d76e9ce-07d0-45cb-9a27-807e13c5ecc4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:31.603432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c6b6ab-6578-4b94-9236-5a8dc8c29cef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:31.604048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c6b6ab-6578-4b94-9236-5a8dc8c29cef admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.604332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c6b6ab-6578-4b94-9236-5a8dc8c29cef admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.604650 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c6b6ab-6578-4b94-9236-5a8dc8c29cef admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.605330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c6b6ab-6578-4b94-9236-5a8dc8c29cef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:31.606916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c6b6ab-6578-4b94-9236-5a8dc8c29cef admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:31.622174 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1132/2246] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:50:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:50:31.623497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e141c6-1ce8-4259-b8ae-7830e8f93967 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:31.634346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e141c6-1ce8-4259-b8ae-7830e8f93967 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:31.641927 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e29ed2-4fba-4fea-8368-d7d9da91995c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:31.642606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e29ed2-4fba-4fea-8368-d7d9da91995c admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.642899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e29ed2-4fba-4fea-8368-d7d9da91995c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.643196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e29ed2-4fba-4fea-8368-d7d9da91995c admin admin] PATH_INFO: `/v3/projects/47ced6e1194347648e62418a5bf0ce03/users/ac5da1e81a2f485580aebeafecb5ecc0/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.659831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e29ed2-4fba-4fea-8368-d7d9da91995c admin admin] RBAC: Authorizing `identity:create_grant(project_id=47ced6e1194347648e62418a5bf0ce03, user_id=ac5da1e81a2f485580aebeafecb5ecc0, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:31.662581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e29ed2-4fba-4fea-8368-d7d9da91995c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:31.676925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1115/2247] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:50:31 2022] PUT /identity/v3/projects/47ced6e1194347648e62418a5bf0ce03/users/ac5da1e81a2f485580aebeafecb5ecc0/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:50:31.687329 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f2418e-c03c-4af3-9f7c-9b8d186942a4 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.687623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f2418e-c03c-4af3-9f7c-9b8d186942a4 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.687907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f2418e-c03c-4af3-9f7c-9b8d186942a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.715814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d6f2418e-c03c-4af3-9f7c-9b8d186942a4 None None] MFA Rules not processed for user `ac5da1e81a2f485580aebeafecb5ecc0`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:31.717731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d6f2418e-c03c-4af3-9f7c-9b8d186942a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:31.764005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:31.785794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1133/2248] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:31 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:31.798121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e29ed2-4fba-4fea-8368-d7d9da91995c admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:31.799963 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-18e29ed2-4fba-4fea-8368-d7d9da91995c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:31.866430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e29ed2-4fba-4fea-8368-d7d9da91995c admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:31.870300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4313ad2f-9cce-4dd9-9eb9-3216660efb19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:31.871056 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313ad2f-9cce-4dd9-9eb9-3216660efb19 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:31.871531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313ad2f-9cce-4dd9-9eb9-3216660efb19 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:31.871816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313ad2f-9cce-4dd9-9eb9-3216660efb19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:31.881638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4313ad2f-9cce-4dd9-9eb9-3216660efb19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:31.883572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4313ad2f-9cce-4dd9-9eb9-3216660efb19 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:31.893107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1116/2249] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:50:31 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:50:32.675764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f2418e-c03c-4af3-9f7c-9b8d186942a4 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:32.677348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d6f2418e-c03c-4af3-9f7c-9b8d186942a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:32.744158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f2418e-c03c-4af3-9f7c-9b8d186942a4 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:32.749321 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:32.749966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:32.750252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:32.750530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] PATH_INFO: `/v3/users/e758990a520b45a0b5c294972f0dd37a` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:32.751848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e758990a520b45a0b5c294972f0dd37a)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:32.753990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:32.799464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e758990a520b45a0b5c294972f0dd37a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:32.800108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e758990a520b45a0b5c294972f0dd37a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:32.800739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e758990a520b45a0b5c294972f0dd37a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:32.813332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e758990a520b45a0b5c294972f0dd37a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:32.819941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e758990a520b45a0b5c294972f0dd37a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:32.832401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1134/2250] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:50:32 2022] DELETE /identity/v3/users/e758990a520b45a0b5c294972f0dd37a => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:32.837426 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e858b726-173e-4122-86be-bfc5da917a80 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:32.837937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e858b726-173e-4122-86be-bfc5da917a80 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:32.838030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e858b726-173e-4122-86be-bfc5da917a80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:32.857094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e858b726-173e-4122-86be-bfc5da917a80 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:32.858876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e858b726-173e-4122-86be-bfc5da917a80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:32.882746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:32.919455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1117/2251] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:50:32.929733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:32.931163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:32.988155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ffd4d09-b7c6-4678-951f-e3b130fc5390 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:32.991843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40cd9e8-1e7b-41e7-8b64-a1c0189fe4c0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:32.992461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40cd9e8-1e7b-41e7-8b64-a1c0189fe4c0 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:32.992787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40cd9e8-1e7b-41e7-8b64-a1c0189fe4c0 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:32.993071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40cd9e8-1e7b-41e7-8b64-a1c0189fe4c0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:33.002675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40cd9e8-1e7b-41e7-8b64-a1c0189fe4c0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:33.004101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40cd9e8-1e7b-41e7-8b64-a1c0189fe4c0 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:33.013507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1135/2252] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:50:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:50:33.179393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e858b726-173e-4122-86be-bfc5da917a80 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:33.180914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e858b726-173e-4122-86be-bfc5da917a80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:33.195437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e858b726-173e-4122-86be-bfc5da917a80 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:33.201301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:33.202265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:33.202634 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:33.202973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] PATH_INFO: `/v3/projects/48e14495c8f7484ab3adc511e812d3e8` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:33.204152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=48e14495c8f7484ab3adc511e812d3e8)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:33.206110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:33.253875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48e14495c8f7484ab3adc511e812d3e8', 'request_id': 'req-1b6af90d-e848-47ae-9e34-6cb7ff298e58'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:33.260762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48e14495c8f7484ab3adc511e812d3e8', 'request_id': 'req-1b6af90d-e848-47ae-9e34-6cb7ff298e58'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:33.266584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48e14495c8f7484ab3adc511e812d3e8', 'request_id': 'req-1b6af90d-e848-47ae-9e34-6cb7ff298e58'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:33.277497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] The token cache is being invalidate because project 48e14495c8f7484ab3adc511e812d3e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:50:33.277788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:33.278429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:33.280763 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1118/2253] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:50:33 2022] DELETE /identity/v3/projects/48e14495c8f7484ab3adc511e812d3e8 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:50:37.241856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafcfaf8-9606-46c9-adbf-6ea5ed608ce5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:37.242467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafcfaf8-9606-46c9-adbf-6ea5ed608ce5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:37.242467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafcfaf8-9606-46c9-adbf-6ea5ed608ce5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:37.262019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-aafcfaf8-9606-46c9-adbf-6ea5ed608ce5 None None] MFA Rules not processed for user `5f9d45265c6f47609f9f8abcf3526ff0`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:37.263245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-aafcfaf8-9606-46c9-adbf-6ea5ed608ce5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:37.268196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:37.321690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1136/2254] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:37 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:42.250426 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:42.251855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:42.316206 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6af90d-e848-47ae-9e34-6cb7ff298e58 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:42.320909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:42.321544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:42.321830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:42.322120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] PATH_INFO: `/v3/users/ac5da1e81a2f485580aebeafecb5ecc0` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:42.323429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac5da1e81a2f485580aebeafecb5ecc0)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:42.325814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:42.373850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac5da1e81a2f485580aebeafecb5ecc0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:42.380559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac5da1e81a2f485580aebeafecb5ecc0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:42.381203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac5da1e81a2f485580aebeafecb5ecc0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:42.381839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac5da1e81a2f485580aebeafecb5ecc0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:42.383036 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:42.391170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac5da1e81a2f485580aebeafecb5ecc0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:42.409244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1119/2255] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:50:42 2022] DELETE /identity/v3/users/ac5da1e81a2f485580aebeafecb5ecc0 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:42.413710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d25f07-e0bd-4732-9a1f-767fc2e0f804 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:42.414186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d25f07-e0bd-4732-9a1f-767fc2e0f804 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:42.414281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d25f07-e0bd-4732-9a1f-767fc2e0f804 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:42.433779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-71d25f07-e0bd-4732-9a1f-767fc2e0f804 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:42.434999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-71d25f07-e0bd-4732-9a1f-767fc2e0f804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:42.506622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1137/2256] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:42.517008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:42.518580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:42.580051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50044b5f-07ff-4aeb-874b-2bb00a594e53 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:42.583850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d0b9c0-91ec-4626-b8fa-291bd97e2db1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:42.584474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d0b9c0-91ec-4626-b8fa-291bd97e2db1 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:42.584795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d0b9c0-91ec-4626-b8fa-291bd97e2db1 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:42.585083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d0b9c0-91ec-4626-b8fa-291bd97e2db1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:42.595684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d0b9c0-91ec-4626-b8fa-291bd97e2db1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:42.597267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d0b9c0-91ec-4626-b8fa-291bd97e2db1 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:42.611266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1120/2257] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:50:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:50:42.876315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d25f07-e0bd-4732-9a1f-767fc2e0f804 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:42.877791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-71d25f07-e0bd-4732-9a1f-767fc2e0f804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:42.892967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d25f07-e0bd-4732-9a1f-767fc2e0f804 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:42.898877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:42.899698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:42.900176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:42.900633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] PATH_INFO: `/v3/projects/47ced6e1194347648e62418a5bf0ce03` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:42.902524 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=47ced6e1194347648e62418a5bf0ce03)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:42.905407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:42.953749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '47ced6e1194347648e62418a5bf0ce03', 'request_id': 'req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:42.959836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '47ced6e1194347648e62418a5bf0ce03', 'request_id': 'req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:42.966881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '47ced6e1194347648e62418a5bf0ce03', 'request_id': 'req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:42.977298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] The token cache is being invalidate because project 47ced6e1194347648e62418a5bf0ce03 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:50:42.977592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:42.978222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c1a1b2b8-725f-4d17-93a4-5064a862cbc0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:50:42.979727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1138/2258] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:50:42 2022] DELETE /identity/v3/projects/47ced6e1194347648e62418a5bf0ce03 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:42.984942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971228e9-87e6-4bb0-84bc-2bdcfc898af2 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:42.985263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971228e9-87e6-4bb0-84bc-2bdcfc898af2 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:42.985548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971228e9-87e6-4bb0-84bc-2bdcfc898af2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:43.008251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-971228e9-87e6-4bb0-84bc-2bdcfc898af2 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:43.009416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-971228e9-87e6-4bb0-84bc-2bdcfc898af2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:43.067292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1121/2259] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:43.075066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d6f4e2-e624-4264-bb9d-8fb1c2fb2c64 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:43.075372 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d6f4e2-e624-4264-bb9d-8fb1c2fb2c64 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:43.075665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d6f4e2-e624-4264-bb9d-8fb1c2fb2c64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:43.096618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a1d6f4e2-e624-4264-bb9d-8fb1c2fb2c64 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:43.097869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a1d6f4e2-e624-4264-bb9d-8fb1c2fb2c64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:43.105806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1139/2260] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:43.109894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971228e9-87e6-4bb0-84bc-2bdcfc898af2 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:43.119438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971228e9-87e6-4bb0-84bc-2bdcfc898af2 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:43.124574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154930ea-86bd-4b41-9039-ac3d48f9eab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:43.125206 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154930ea-86bd-4b41-9039-ac3d48f9eab3 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:43.125487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154930ea-86bd-4b41-9039-ac3d48f9eab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:43.125762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154930ea-86bd-4b41-9039-ac3d48f9eab3 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:43.126453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154930ea-86bd-4b41-9039-ac3d48f9eab3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:50:43.126757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154930ea-86bd-4b41-9039-ac3d48f9eab3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:43.128332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-154930ea-86bd-4b41-9039-ac3d48f9eab3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:43.140388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1122/2261] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:50:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:50:43.144564 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d6f4e2-e624-4264-bb9d-8fb1c2fb2c64 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:43.153742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d6f4e2-e624-4264-bb9d-8fb1c2fb2c64 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:43.158716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1a74d6-aedb-405a-a8b9-2f56c635dea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:43.159319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a74d6-aedb-405a-a8b9-2f56c635dea7 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:43.159603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a74d6-aedb-405a-a8b9-2f56c635dea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:43.159878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a74d6-aedb-405a-a8b9-2f56c635dea7 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:43.160535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1a74d6-aedb-405a-a8b9-2f56c635dea7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:43.162455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1a74d6-aedb-405a-a8b9-2f56c635dea7 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:43.189942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1140/2262] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:50:43 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:50:43.194505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154930ea-86bd-4b41-9039-ac3d48f9eab3 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:43.204907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154930ea-86bd-4b41-9039-ac3d48f9eab3 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:43.261103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f512cbbb-2e68-48ff-92bb-db0348fd2701 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:43.261713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f512cbbb-2e68-48ff-92bb-db0348fd2701 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:43.261910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f512cbbb-2e68-48ff-92bb-db0348fd2701 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:43.262189 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f512cbbb-2e68-48ff-92bb-db0348fd2701 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:43.262859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f512cbbb-2e68-48ff-92bb-db0348fd2701 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:43.265343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f512cbbb-2e68-48ff-92bb-db0348fd2701 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:43.273921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:43.287312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1123/2263] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:50:43 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:50:43.291649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1a74d6-aedb-405a-a8b9-2f56c635dea7 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:43.302714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1a74d6-aedb-405a-a8b9-2f56c635dea7 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:43.307435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7e8027-7507-457f-8425-20dae069383e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:43.308050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7e8027-7507-457f-8425-20dae069383e admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:43.308342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7e8027-7507-457f-8425-20dae069383e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:43.308656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7e8027-7507-457f-8425-20dae069383e admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:43.309362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7e8027-7507-457f-8425-20dae069383e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:43.311694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7e8027-7507-457f-8425-20dae069383e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:43.316851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:43.323470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1141/2264] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:50:43.327638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f512cbbb-2e68-48ff-92bb-db0348fd2701 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:43.337566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f512cbbb-2e68-48ff-92bb-db0348fd2701 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:43.342695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2948c4e-d090-4e81-8b32-210a97fa811a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:43.343301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2948c4e-d090-4e81-8b32-210a97fa811a admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:43.343584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2948c4e-d090-4e81-8b32-210a97fa811a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:43.343860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2948c4e-d090-4e81-8b32-210a97fa811a admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:43.344554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2948c4e-d090-4e81-8b32-210a97fa811a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:43.346173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2948c4e-d090-4e81-8b32-210a97fa811a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:43.356628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1124/2265] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:50:43 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:50:43.361230 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7e8027-7507-457f-8425-20dae069383e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:43.371118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7e8027-7507-457f-8425-20dae069383e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:43.376890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f072bc-2134-4c0e-84b8-ce9a9df0ca76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:43.377596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f072bc-2134-4c0e-84b8-ce9a9df0ca76 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:43.377908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f072bc-2134-4c0e-84b8-ce9a9df0ca76 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:43.378221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f072bc-2134-4c0e-84b8-ce9a9df0ca76 admin admin] PATH_INFO: `/v3/projects/21c0f0b9063948a9a0183b528bc21130/users/11f44b37b46b4793944285bee7b8427e/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:43.394475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f072bc-2134-4c0e-84b8-ce9a9df0ca76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=21c0f0b9063948a9a0183b528bc21130, user_id=11f44b37b46b4793944285bee7b8427e, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:43.397211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f072bc-2134-4c0e-84b8-ce9a9df0ca76 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:43.410087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1142/2266] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:50:43 2022] PUT /identity/v3/projects/21c0f0b9063948a9a0183b528bc21130/users/11f44b37b46b4793944285bee7b8427e/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:43.415250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c98501d-a89d-46ce-822f-94ae17691734 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:43.415541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c98501d-a89d-46ce-822f-94ae17691734 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:43.415821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c98501d-a89d-46ce-822f-94ae17691734 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:43.435205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-7c98501d-a89d-46ce-822f-94ae17691734 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:43.437208 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7c98501d-a89d-46ce-822f-94ae17691734 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:43.496014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1125/2267] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:43.506154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f072bc-2134-4c0e-84b8-ce9a9df0ca76 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:43.507995 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-92f072bc-2134-4c0e-84b8-ce9a9df0ca76 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:43.564964 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f072bc-2134-4c0e-84b8-ce9a9df0ca76 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:43.568631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ea55d-70d8-4114-a079-6bc2fad2c160 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:43.569248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ea55d-70d8-4114-a079-6bc2fad2c160 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:43.569544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ea55d-70d8-4114-a079-6bc2fad2c160 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:43.569823 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ea55d-70d8-4114-a079-6bc2fad2c160 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:43.578781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2ea55d-70d8-4114-a079-6bc2fad2c160 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:43.580212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2ea55d-70d8-4114-a079-6bc2fad2c160 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:43.589565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1143/2268] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:50:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:50:48.141311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f719d356-73cd-4167-9d06-78dedd3676ad None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:48.141878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f719d356-73cd-4167-9d06-78dedd3676ad None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:48.141878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f719d356-73cd-4167-9d06-78dedd3676ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:48.175679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f719d356-73cd-4167-9d06-78dedd3676ad None None] MFA Rules not processed for user `11f44b37b46b4793944285bee7b8427e`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:48.176962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f719d356-73cd-4167-9d06-78dedd3676ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:48.252775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1126/2269] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:48.257346 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ea55d-70d8-4114-a079-6bc2fad2c160 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:48.268071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ea55d-70d8-4114-a079-6bc2fad2c160 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:48.272748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb15639-c232-4be2-b6e9-aeb54095228e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:48.273384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb15639-c232-4be2-b6e9-aeb54095228e admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:48.273667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb15639-c232-4be2-b6e9-aeb54095228e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:48.273941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb15639-c232-4be2-b6e9-aeb54095228e admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:48.275161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb15639-c232-4be2-b6e9-aeb54095228e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:48.277121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb15639-c232-4be2-b6e9-aeb54095228e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:48.307379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1144/2270] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:50:48 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:50:48.311444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f719d356-73cd-4167-9d06-78dedd3676ad None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:48.322762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f719d356-73cd-4167-9d06-78dedd3676ad None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:48.414592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebefca04-d176-42d2-a255-84e55250f5e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:48.415232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebefca04-d176-42d2-a255-84e55250f5e5 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:48.415507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebefca04-d176-42d2-a255-84e55250f5e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:48.415785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebefca04-d176-42d2-a255-84e55250f5e5 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:48.416454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebefca04-d176-42d2-a255-84e55250f5e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:48.418442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebefca04-d176-42d2-a255-84e55250f5e5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:48.438408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:48.444233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1127/2271] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:50:48 2022] POST /identity/v3/users => generated 299 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:50:48.448442 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb15639-c232-4be2-b6e9-aeb54095228e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:48.461038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb15639-c232-4be2-b6e9-aeb54095228e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:48.468421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b7ee9b-05d7-4037-8bea-85a9c83ac6f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:48.469075 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b7ee9b-05d7-4037-8bea-85a9c83ac6f0 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:48.469508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b7ee9b-05d7-4037-8bea-85a9c83ac6f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:48.469797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b7ee9b-05d7-4037-8bea-85a9c83ac6f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:48.470551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b7ee9b-05d7-4037-8bea-85a9c83ac6f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:48.472360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b7ee9b-05d7-4037-8bea-85a9c83ac6f0 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:48.492709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1145/2272] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:50:48 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:50:48.495279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebefca04-d176-42d2-a255-84e55250f5e5 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:48.507197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebefca04-d176-42d2-a255-84e55250f5e5 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:48.512022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9172e19-6c08-4b20-8974-5d449126bcb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:48.512737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9172e19-6c08-4b20-8974-5d449126bcb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:48.513024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9172e19-6c08-4b20-8974-5d449126bcb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:48.513855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9172e19-6c08-4b20-8974-5d449126bcb2 admin admin] PATH_INFO: `/v3/projects/824084791d65402f8a85aba10357e892/users/d0548a60546d4ae581a9dd32320a3a57/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:48.535802 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9172e19-6c08-4b20-8974-5d449126bcb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=824084791d65402f8a85aba10357e892, user_id=d0548a60546d4ae581a9dd32320a3a57, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:48.538592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9172e19-6c08-4b20-8974-5d449126bcb2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:48.554194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1128/2273] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:50:48 2022] PUT /identity/v3/projects/824084791d65402f8a85aba10357e892/users/d0548a60546d4ae581a9dd32320a3a57/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:48.557875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b7ee9b-05d7-4037-8bea-85a9c83ac6f0 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:48.569045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b7ee9b-05d7-4037-8bea-85a9c83ac6f0 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:48.659826 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5773a1d-8ed3-419d-86b7-8d56df360ae8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:48.660479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5773a1d-8ed3-419d-86b7-8d56df360ae8 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:48.660813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5773a1d-8ed3-419d-86b7-8d56df360ae8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:48.668715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5773a1d-8ed3-419d-86b7-8d56df360ae8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:48.670701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5773a1d-8ed3-419d-86b7-8d56df360ae8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:48.672724 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5773a1d-8ed3-419d-86b7-8d56df360ae8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:48.687966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1146/2274] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:50:48 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:50:48.689298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9172e19-6c08-4b20-8974-5d449126bcb2 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:48.701773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9172e19-6c08-4b20-8974-5d449126bcb2 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:48.709920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45a81d-374c-4d10-83a7-34930f82785a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:48.710707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45a81d-374c-4d10-83a7-34930f82785a admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:48.711090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45a81d-374c-4d10-83a7-34930f82785a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:48.711652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d45a81d-374c-4d10-83a7-34930f82785a admin admin] PATH_INFO: `/v3/projects/824084791d65402f8a85aba10357e892/users/d0548a60546d4ae581a9dd32320a3a57/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:48.713940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d45a81d-374c-4d10-83a7-34930f82785a admin admin] RBAC: Authorizing `identity:create_grant(project_id=824084791d65402f8a85aba10357e892, user_id=d0548a60546d4ae581a9dd32320a3a57, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:48.718352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d45a81d-374c-4d10-83a7-34930f82785a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:48.730622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1129/2275] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:50:48 2022] PUT /identity/v3/projects/824084791d65402f8a85aba10357e892/users/d0548a60546d4ae581a9dd32320a3a57/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:50:53.886916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc422745-838c-42c5-b712-ea3c722cbff5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:53.887458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc422745-838c-42c5-b712-ea3c722cbff5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:53.887458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc422745-838c-42c5-b712-ea3c722cbff5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:53.922171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-bc422745-838c-42c5-b712-ea3c722cbff5 None None] MFA Rules not processed for user `d0548a60546d4ae581a9dd32320a3a57`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:53.923436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-bc422745-838c-42c5-b712-ea3c722cbff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:53.961689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:50:54.011717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1147/2276] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:50:53 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:54.026631 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45a81d-374c-4d10-83a7-34930f82785a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:54.028806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4d45a81d-374c-4d10-83a7-34930f82785a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:54.107090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d45a81d-374c-4d10-83a7-34930f82785a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:54.113423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e820a34-cb0c-4a8e-85d7-92e844e0ef9b service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3749dc021aac478f88072ced48f4cee4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:54.114336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e820a34-cb0c-4a8e-85d7-92e844e0ef9b service glance] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:54.114770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e820a34-cb0c-4a8e-85d7-92e844e0ef9b service glance] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:54.115254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e820a34-cb0c-4a8e-85d7-92e844e0ef9b service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:54.193054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e820a34-cb0c-4a8e-85d7-92e844e0ef9b service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:54.194524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e820a34-cb0c-4a8e-85d7-92e844e0ef9b service glance] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:54.206207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1130/2277] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:50:54 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:50:54.352254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778a623-db34-44e4-9b8a-5516a561bed2 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:54.352560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778a623-db34-44e4-9b8a-5516a561bed2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:54.352878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0778a623-db34-44e4-9b8a-5516a561bed2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:54.387803 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-0778a623-db34-44e4-9b8a-5516a561bed2 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:50:54.392113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0778a623-db34-44e4-9b8a-5516a561bed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:54.470730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1148/2278] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:50:54 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:50:54.495971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e820a34-cb0c-4a8e-85d7-92e844e0ef9b service glance] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:50:54.497607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5e820a34-cb0c-4a8e-85d7-92e844e0ef9b service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:50:54.578641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e820a34-cb0c-4a8e-85d7-92e844e0ef9b service glance] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:50:54.601096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3201ef37-7d93-4000-9083-fc6e2a7c6909 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:50:54.601876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3201ef37-7d93-4000-9083-fc6e2a7c6909 service swift] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:50:54.602137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3201ef37-7d93-4000-9083-fc6e2a7c6909 service swift] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:50:54.602380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3201ef37-7d93-4000-9083-fc6e2a7c6909 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:50:54.620093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3201ef37-7d93-4000-9083-fc6e2a7c6909 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:50:54.621832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3201ef37-7d93-4000-9083-fc6e2a7c6909 service swift] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:50:54.631025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1131/2279] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:50:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:51:03.743710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbee03f1-0203-4159-8006-dd73da66e144 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:51:03.744236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbee03f1-0203-4159-8006-dd73da66e144 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:51:03.744348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbee03f1-0203-4159-8006-dd73da66e144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:51:03.762809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-cbee03f1-0203-4159-8006-dd73da66e144 None None] MFA Rules not processed for user `11f44b37b46b4793944285bee7b8427e`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:51:03.763998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-cbee03f1-0203-4159-8006-dd73da66e144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:51:03.770794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1149/2280] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:51:03 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:51:04.628112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:51:05.792821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4617555-290e-4b28-a2e0-18f62276bdf9 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:51:05.793392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4617555-290e-4b28-a2e0-18f62276bdf9 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:51:05.793392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4617555-290e-4b28-a2e0-18f62276bdf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:51:05.814162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f4617555-290e-4b28-a2e0-18f62276bdf9 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:51:05.815418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f4617555-290e-4b28-a2e0-18f62276bdf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:51:05.822709 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1132/2281] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:51:05 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:51:05.836509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbee03f1-0203-4159-8006-dd73da66e144 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:51:05.849245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbee03f1-0203-4159-8006-dd73da66e144 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:51:05.854384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef6ae07-2fc4-4e45-88dd-dc30e96a5f2b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:51:05.855000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef6ae07-2fc4-4e45-88dd-dc30e96a5f2b service swift] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:51:05.855291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef6ae07-2fc4-4e45-88dd-dc30e96a5f2b service swift] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:51:05.855563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef6ae07-2fc4-4e45-88dd-dc30e96a5f2b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:51:05.864259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef6ae07-2fc4-4e45-88dd-dc30e96a5f2b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:51:05.865869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef6ae07-2fc4-4e45-88dd-dc30e96a5f2b service swift] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:51:05.874568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1150/2282] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:51:05 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 29 04:51:15.808149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:51:15.870718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=49 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:51:38.526934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f027d75-ea3f-4f63-a431-f1336fc86687 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:51:38.527456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f027d75-ea3f-4f63-a431-f1336fc86687 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:51:38.527550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f027d75-ea3f-4f63-a431-f1336fc86687 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:51:38.546816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6f027d75-ea3f-4f63-a431-f1336fc86687 None None] MFA Rules not processed for user `572408fd157844dcae13000d8e9c4b7a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:51:38.548014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6f027d75-ea3f-4f63-a431-f1336fc86687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:51:38.600503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1133/2283] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:51:38 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:51:48.594163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=25 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:51:55.918359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef6ae07-2fc4-4e45-88dd-dc30e96a5f2b service swift] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:51:55.920418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-cef6ae07-2fc4-4e45-88dd-dc30e96a5f2b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:51:55.989413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef6ae07-2fc4-4e45-88dd-dc30e96a5f2b service swift] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:51:55.993897 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1675eaab-6668-4359-ab87-3119e7da4d0b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:51:55.994791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1675eaab-6668-4359-ab87-3119e7da4d0b service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:51:55.995083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1675eaab-6668-4359-ab87-3119e7da4d0b service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:51:55.995367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1675eaab-6668-4359-ab87-3119e7da4d0b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:51:56.005866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1675eaab-6668-4359-ab87-3119e7da4d0b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:51:56.007312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1675eaab-6668-4359-ab87-3119e7da4d0b service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:51:56.023879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1151/2284] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:51:55 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:52:06.016572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=26 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:52:19.760291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f027d75-ea3f-4f63-a431-f1336fc86687 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:19.771854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f027d75-ea3f-4f63-a431-f1336fc86687 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:19.775820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728a5a8a-2fcc-49e7-ab4f-55151860ca83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:19.776921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728a5a8a-2fcc-49e7-ab4f-55151860ca83 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:19.777519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728a5a8a-2fcc-49e7-ab4f-55151860ca83 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:19.777519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728a5a8a-2fcc-49e7-ab4f-55151860ca83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:19.780828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-728a5a8a-2fcc-49e7-ab4f-55151860ca83 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:19.869525 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728a5a8a-2fcc-49e7-ab4f-55151860ca83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:19.871747 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728a5a8a-2fcc-49e7-ab4f-55151860ca83 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:19.884165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1134/2285] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:52:19 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:52:21.069897 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1675eaab-6668-4359-ab87-3119e7da4d0b service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:21.079337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1675eaab-6668-4359-ab87-3119e7da4d0b service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:21.082968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a1756a-72a8-4e83-9141-cd1652666f44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:21.083573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a1756a-72a8-4e83-9141-cd1652666f44 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:21.083862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a1756a-72a8-4e83-9141-cd1652666f44 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:21.084144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a1756a-72a8-4e83-9141-cd1652666f44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:21.085904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-59a1756a-72a8-4e83-9141-cd1652666f44 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:21.099890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a1756a-72a8-4e83-9141-cd1652666f44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:21.101348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a1756a-72a8-4e83-9141-cd1652666f44 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:21.112509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1152/2286] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:52:21 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:52:29.878164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=34 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:52:31.108209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:52:35.722873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6b28cf-cd47-42e9-8393-c705d47d34bd None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:35.723368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6b28cf-cd47-42e9-8393-c705d47d34bd None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:35.723483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6b28cf-cd47-42e9-8393-c705d47d34bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:35.743438 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-3d6b28cf-cd47-42e9-8393-c705d47d34bd None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:52:35.744720 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3d6b28cf-cd47-42e9-8393-c705d47d34bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:35.752030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1135/2287] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:52:35 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:52:35.792875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a1756a-72a8-4e83-9141-cd1652666f44 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:35.803995 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a1756a-72a8-4e83-9141-cd1652666f44 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:35.809538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ddb12d-9097-41b7-9531-a59d3d943220 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:35.810147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ddb12d-9097-41b7-9531-a59d3d943220 service swift] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:35.810433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ddb12d-9097-41b7-9531-a59d3d943220 service swift] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:35.810763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ddb12d-9097-41b7-9531-a59d3d943220 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:35.819658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ddb12d-9097-41b7-9531-a59d3d943220 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:35.821268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ddb12d-9097-41b7-9531-a59d3d943220 service swift] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:35.830080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1153/2288] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:52:35 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:52:45.737566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:52:45.826822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:52:46.924482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6b28cf-cd47-42e9-8393-c705d47d34bd None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:46.934282 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6b28cf-cd47-42e9-8393-c705d47d34bd None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:46.937791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10b7bbf-8c50-48b1-b95a-05c4d26e4609 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:46.938399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b7bbf-8c50-48b1-b95a-05c4d26e4609 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:46.938681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b7bbf-8c50-48b1-b95a-05c4d26e4609 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:46.938957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10b7bbf-8c50-48b1-b95a-05c4d26e4609 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:46.940754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d10b7bbf-8c50-48b1-b95a-05c4d26e4609 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:46.950341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10b7bbf-8c50-48b1-b95a-05c4d26e4609 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:46.951786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10b7bbf-8c50-48b1-b95a-05c4d26e4609 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:46.959867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1136/2289] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:52:46 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:52:50.084750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ddb12d-9097-41b7-9531-a59d3d943220 service swift] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:50.094161 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ddb12d-9097-41b7-9531-a59d3d943220 service swift] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:50.097808 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90d1910-1684-4044-ad2e-92394cf6eb01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:50.098414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90d1910-1684-4044-ad2e-92394cf6eb01 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.098705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90d1910-1684-4044-ad2e-92394cf6eb01 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.098989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90d1910-1684-4044-ad2e-92394cf6eb01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.100781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b90d1910-1684-4044-ad2e-92394cf6eb01 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:50.110201 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90d1910-1684-4044-ad2e-92394cf6eb01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:50.111621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90d1910-1684-4044-ad2e-92394cf6eb01 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:50.119501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1154/2290] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:52:50 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:52:50.244289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c56d6f9-f86d-4f4d-9d22-3267cccb8d56 None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.244879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c56d6f9-f86d-4f4d-9d22-3267cccb8d56 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.245118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c56d6f9-f86d-4f4d-9d22-3267cccb8d56 None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.246580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1137/2291] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:52:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:52:50.252686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad0b67c-338c-4b04-a4e8-98e5461441d0 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.253044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad0b67c-338c-4b04-a4e8-98e5461441d0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.253334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad0b67c-338c-4b04-a4e8-98e5461441d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.274205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-cad0b67c-338c-4b04-a4e8-98e5461441d0 None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:52:50.275443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-cad0b67c-338c-4b04-a4e8-98e5461441d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:50.285504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1155/2292] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:52:50 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:52:50.297915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c56d6f9-f86d-4f4d-9d22-3267cccb8d56 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:50.309734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c56d6f9-f86d-4f4d-9d22-3267cccb8d56 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:50.313675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87397f26-4aca-4d58-a508-035a797b0b5d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3749dc021aac478f88072ced48f4cee4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:50.314299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87397f26-4aca-4d58-a508-035a797b0b5d service glance] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.314643 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87397f26-4aca-4d58-a508-035a797b0b5d service glance] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.314946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87397f26-4aca-4d58-a508-035a797b0b5d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.325741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87397f26-4aca-4d58-a508-035a797b0b5d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:50.327401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87397f26-4aca-4d58-a508-035a797b0b5d service glance] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:50.338698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1138/2293] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:52:50 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:52:50.410087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac71e48a-50c4-4948-a8ad-3a0c34a2493d None None] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.410581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac71e48a-50c4-4948-a8ad-3a0c34a2493d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.411033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac71e48a-50c4-4948-a8ad-3a0c34a2493d None None] PATH_INFO: `/` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.412660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1156/2294] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:52:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:52:50.417514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f25a808-6ade-428e-9398-b226a242b698 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.417806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f25a808-6ade-428e-9398-b226a242b698 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.418105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f25a808-6ade-428e-9398-b226a242b698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.438969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-2f25a808-6ade-428e-9398-b226a242b698 None None] MFA Rules not processed for user `f6726ef0c4184040a18a76898bb738d1`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:52:50.440229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2f25a808-6ade-428e-9398-b226a242b698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:50.446870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1139/2295] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:52:50 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:52:50.458612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac71e48a-50c4-4948-a8ad-3a0c34a2493d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:50.469777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac71e48a-50c4-4948-a8ad-3a0c34a2493d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:50.473404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aabc230-08ab-4d13-a60f-60ff0daf8279 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:50.474034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aabc230-08ab-4d13-a60f-60ff0daf8279 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.474316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aabc230-08ab-4d13-a60f-60ff0daf8279 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.474592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aabc230-08ab-4d13-a60f-60ff0daf8279 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.481461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aabc230-08ab-4d13-a60f-60ff0daf8279 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:50.483491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aabc230-08ab-4d13-a60f-60ff0daf8279 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:50.491582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1157/2296] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:52:50 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:52:50.577528 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26afea5-a8a6-4d79-aa19-92993d4542ed None None] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.577821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26afea5-a8a6-4d79-aa19-92993d4542ed None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.578106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26afea5-a8a6-4d79-aa19-92993d4542ed None None] PATH_INFO: `/` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.579466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1140/2297] 10.222.0.40 () {54 vars in 921 bytes} [Fri Jul 29 04:52:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 29 04:52:50.584720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2986d794-5630-40ac-84b6-31e337d0c6f2 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.585014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2986d794-5630-40ac-84b6-31e337d0c6f2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.585298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2986d794-5630-40ac-84b6-31e337d0c6f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.605192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2986d794-5630-40ac-84b6-31e337d0c6f2 None None] MFA Rules not processed for user `58c33a72a871432b88da4f0bf1bc514c`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:52:50.606334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2986d794-5630-40ac-84b6-31e337d0c6f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:50.615963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1158/2298] 10.222.0.40 () {58 vars in 1018 bytes} [Fri Jul 29 04:52:50 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:52:50.626882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26afea5-a8a6-4d79-aa19-92993d4542ed None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:50.637878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26afea5-a8a6-4d79-aa19-92993d4542ed None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:50.641809 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8458b4-692a-401d-9899-86e91e2b45ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:50.642469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8458b4-692a-401d-9899-86e91e2b45ee service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.642735 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8458b4-692a-401d-9899-86e91e2b45ee service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.643018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8458b4-692a-401d-9899-86e91e2b45ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.654561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8458b4-692a-401d-9899-86e91e2b45ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:50.656052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8458b4-692a-401d-9899-86e91e2b45ee service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:50.667729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1141/2299] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:52:50 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:52:50.734007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2986d794-5630-40ac-84b6-31e337d0c6f2 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:50.735468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2986d794-5630-40ac-84b6-31e337d0c6f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:50.805800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2986d794-5630-40ac-84b6-31e337d0c6f2 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:50.810949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665703b1-462c-44c7-b570-0a66d5a58413 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:50.811568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665703b1-462c-44c7-b570-0a66d5a58413 service placement] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:50.811859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665703b1-462c-44c7-b570-0a66d5a58413 service placement] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:50.812140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665703b1-462c-44c7-b570-0a66d5a58413 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:50.813966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-665703b1-462c-44c7-b570-0a66d5a58413 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:50.825159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665703b1-462c-44c7-b570-0a66d5a58413 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:50.826953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665703b1-462c-44c7-b570-0a66d5a58413 service placement] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:50.836710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1159/2300] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:52:50 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:52:53.401029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfb8e7f-cf07-45a1-9b6d-4c70ff0319e1 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:53.401531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfb8e7f-cf07-45a1-9b6d-4c70ff0319e1 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:53.401639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfb8e7f-cf07-45a1-9b6d-4c70ff0319e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:53.421441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-8dfb8e7f-cf07-45a1-9b6d-4c70ff0319e1 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:52:53.422575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8dfb8e7f-cf07-45a1-9b6d-4c70ff0319e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:53.429775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1142/2301] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:52:53 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:52:53.442649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665703b1-462c-44c7-b570-0a66d5a58413 service placement] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:53.454246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665703b1-462c-44c7-b570-0a66d5a58413 service placement] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:53.459527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21909e54-b44f-4230-bee5-687a1286dfce service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:53.460150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21909e54-b44f-4230-bee5-687a1286dfce service swift] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:53.460443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21909e54-b44f-4230-bee5-687a1286dfce service swift] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:53.460765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21909e54-b44f-4230-bee5-687a1286dfce service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:53.468941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21909e54-b44f-4230-bee5-687a1286dfce service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:53.470515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21909e54-b44f-4230-bee5-687a1286dfce service swift] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:53.478511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1160/2302] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:52:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:52:54.320483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfb8e7f-cf07-45a1-9b6d-4c70ff0319e1 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:52:54.321900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8dfb8e7f-cf07-45a1-9b6d-4c70ff0319e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:54.338626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfb8e7f-cf07-45a1-9b6d-4c70ff0319e1 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:52:54.345421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcc3d5a-dfbd-4b54-b623-743b50a5a8f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:52:54.346062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcc3d5a-dfbd-4b54-b623-743b50a5a8f4 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:52:54.346349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcc3d5a-dfbd-4b54-b623-743b50a5a8f4 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:52:54.346629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcc3d5a-dfbd-4b54-b623-743b50a5a8f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:52:54.348319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3dcc3d5a-dfbd-4b54-b623-743b50a5a8f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:52:54.412937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dcc3d5a-dfbd-4b54-b623-743b50a5a8f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:52:54.414952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dcc3d5a-dfbd-4b54-b623-743b50a5a8f4 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:52:54.425850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1143/2303] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:52:54 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:53:03.475269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=56 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:53:04.420764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=52 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:53:21.634411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21909e54-b44f-4230-bee5-687a1286dfce service swift] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:53:21.635927 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-21909e54-b44f-4230-bee5-687a1286dfce service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:53:21.653504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21909e54-b44f-4230-bee5-687a1286dfce service swift] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:53:21.659141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e2d83c-31df-4a42-a7e3-f1b3b543e22b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:53:21.660001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e2d83c-31df-4a42-a7e3-f1b3b543e22b service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:53:21.660001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e2d83c-31df-4a42-a7e3-f1b3b543e22b service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:53:21.660253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e2d83c-31df-4a42-a7e3-f1b3b543e22b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:53:21.664988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-98e2d83c-31df-4a42-a7e3-f1b3b543e22b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:53:21.675870 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e2d83c-31df-4a42-a7e3-f1b3b543e22b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:53:21.677525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e2d83c-31df-4a42-a7e3-f1b3b543e22b service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:53:21.689709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1161/2304] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:53:21 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:53:24.589583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bcb310-b452-485e-8c00-d8e955078180 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:53:24.590066 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bcb310-b452-485e-8c00-d8e955078180 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:53:24.590155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bcb310-b452-485e-8c00-d8e955078180 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:53:24.610629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-74bcb310-b452-485e-8c00-d8e955078180 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:53:24.611892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-74bcb310-b452-485e-8c00-d8e955078180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:53:24.622400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1144/2305] 10.222.0.40 () {58 vars in 967 bytes} [Fri Jul 29 04:53:24 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:53:24.633182 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e2d83c-31df-4a42-a7e3-f1b3b543e22b service nova] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:53:24.645649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e2d83c-31df-4a42-a7e3-f1b3b543e22b service nova] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:53:24.673721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58835db-c68c-49eb-9c13-407cc72cda73 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:53:24.674362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58835db-c68c-49eb-9c13-407cc72cda73 service swift] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:53:24.674651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58835db-c68c-49eb-9c13-407cc72cda73 service swift] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:53:24.674943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58835db-c68c-49eb-9c13-407cc72cda73 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:53:24.683992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58835db-c68c-49eb-9c13-407cc72cda73 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:53:24.686162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58835db-c68c-49eb-9c13-407cc72cda73 service swift] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:53:24.695354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1162/2306] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:53:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:53:34.604787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:53:34.676872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bcb310-b452-485e-8c00-d8e955078180 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:53:34.690082 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bcb310-b452-485e-8c00-d8e955078180 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:53:34.691704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=12 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:53:34.696026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0335dc4d-803c-4b20-b02c-f2d94fa6f7d8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:53:34.696708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335dc4d-803c-4b20-b02c-f2d94fa6f7d8 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:53:34.697019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335dc4d-803c-4b20-b02c-f2d94fa6f7d8 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:53:34.697298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335dc4d-803c-4b20-b02c-f2d94fa6f7d8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:53:34.699050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0335dc4d-803c-4b20-b02c-f2d94fa6f7d8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:53:34.758576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0335dc4d-803c-4b20-b02c-f2d94fa6f7d8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:53:34.760174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0335dc4d-803c-4b20-b02c-f2d94fa6f7d8 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:53:34.770453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1145/2307] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:53:34 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:53:41.861471 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879d710f-7d99-4bcb-8cfd-defc83beb0d0 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:53:41.861974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879d710f-7d99-4bcb-8cfd-defc83beb0d0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:53:41.862074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879d710f-7d99-4bcb-8cfd-defc83beb0d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:53:41.880378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-879d710f-7d99-4bcb-8cfd-defc83beb0d0 None None] MFA Rules not processed for user `5f9d45265c6f47609f9f8abcf3526ff0`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:53:41.881624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-879d710f-7d99-4bcb-8cfd-defc83beb0d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:53:41.888624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1163/2308] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:53:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:53:44.765981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=28 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:53:48.015666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0335dc4d-803c-4b20-b02c-f2d94fa6f7d8 service nova] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:53:48.030822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0335dc4d-803c-4b20-b02c-f2d94fa6f7d8 service nova] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:53:48.047304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c5ddb3-b3c4-4543-81ff-68389b3e8301 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:53:48.047943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c5ddb3-b3c4-4543-81ff-68389b3e8301 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:53:48.048229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c5ddb3-b3c4-4543-81ff-68389b3e8301 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:53:48.048979 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c5ddb3-b3c4-4543-81ff-68389b3e8301 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:53:48.050727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-51c5ddb3-b3c4-4543-81ff-68389b3e8301 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:53:48.066563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c5ddb3-b3c4-4543-81ff-68389b3e8301 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:53:48.068166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c5ddb3-b3c4-4543-81ff-68389b3e8301 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:53:48.079981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1146/2309] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:53:48 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:53:51.875724 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:53:52.757502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879d710f-7d99-4bcb-8cfd-defc83beb0d0 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:53:52.770387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879d710f-7d99-4bcb-8cfd-defc83beb0d0 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:53:52.775818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0c6c65-fa2b-4abe-bee1-55ea9a319254 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:53:52.776435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0c6c65-fa2b-4abe-bee1-55ea9a319254 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:53:52.776768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0c6c65-fa2b-4abe-bee1-55ea9a319254 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:53:52.777053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0c6c65-fa2b-4abe-bee1-55ea9a319254 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:53:52.779105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5a0c6c65-fa2b-4abe-bee1-55ea9a319254 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:53:52.791715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0c6c65-fa2b-4abe-bee1-55ea9a319254 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:53:52.794503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0c6c65-fa2b-4abe-bee1-55ea9a319254 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:53:52.806815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1164/2310] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:53:52 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:53:56.433162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c5ddb3-b3c4-4543-81ff-68389b3e8301 service nova] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:53:56.444777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c5ddb3-b3c4-4543-81ff-68389b3e8301 service nova] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:53:56.452422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2acc72c-7943-4b6d-8ce6-82fc832cf740 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:53:56.453094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2acc72c-7943-4b6d-8ce6-82fc832cf740 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:53:56.453400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2acc72c-7943-4b6d-8ce6-82fc832cf740 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:53:56.453703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2acc72c-7943-4b6d-8ce6-82fc832cf740 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:53:56.455635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c2acc72c-7943-4b6d-8ce6-82fc832cf740 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:53:56.469588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2acc72c-7943-4b6d-8ce6-82fc832cf740 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:53:56.471203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2acc72c-7943-4b6d-8ce6-82fc832cf740 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:53:56.484536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1147/2311] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:53:56 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:54:02.801888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:54:05.883805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6f1850-4af7-4f28-823c-296bf08bf991 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:54:05.884444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6f1850-4af7-4f28-823c-296bf08bf991 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:54:05.884444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6f1850-4af7-4f28-823c-296bf08bf991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:54:05.903657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-cb6f1850-4af7-4f28-823c-296bf08bf991 None None] MFA Rules not processed for user `572408fd157844dcae13000d8e9c4b7a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:54:05.904901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-cb6f1850-4af7-4f28-823c-296bf08bf991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:54:05.911755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1165/2312] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:54:05 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:54:06.211666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2acc72c-7943-4b6d-8ce6-82fc832cf740 service nova] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:54:06.221625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2acc72c-7943-4b6d-8ce6-82fc832cf740 service nova] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:54:06.225230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acab5af-6ba3-4e3e-a6b2-596513dd26fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:54:06.225841 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acab5af-6ba3-4e3e-a6b2-596513dd26fa service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:54:06.226133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acab5af-6ba3-4e3e-a6b2-596513dd26fa service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:54:06.226415 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acab5af-6ba3-4e3e-a6b2-596513dd26fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:54:06.228168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8acab5af-6ba3-4e3e-a6b2-596513dd26fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:54:06.238544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acab5af-6ba3-4e3e-a6b2-596513dd26fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:54:06.239968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acab5af-6ba3-4e3e-a6b2-596513dd26fa service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:54:06.248625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1148/2313] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:54:06 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:54:15.898332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:54:16.244292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=18 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:54:32.907384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6f1850-4af7-4f28-823c-296bf08bf991 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:54:32.920796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6f1850-4af7-4f28-823c-296bf08bf991 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:54:32.926257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc07b718-e210-435b-863e-30b14f585a4d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:54:32.926878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc07b718-e210-435b-863e-30b14f585a4d service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:54:32.927168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc07b718-e210-435b-863e-30b14f585a4d service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:54:32.927451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc07b718-e210-435b-863e-30b14f585a4d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:54:32.929267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fc07b718-e210-435b-863e-30b14f585a4d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:54:32.987489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc07b718-e210-435b-863e-30b14f585a4d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:54:32.989123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc07b718-e210-435b-863e-30b14f585a4d service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:54:32.999048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1166/2314] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:54:32 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:54:41.364380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac22e36-cbf3-46d2-be50-b88f4cf9c87a None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:54:41.364947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac22e36-cbf3-46d2-be50-b88f4cf9c87a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:54:41.365072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac22e36-cbf3-46d2-be50-b88f4cf9c87a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:54:41.385072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4ac22e36-cbf3-46d2-be50-b88f4cf9c87a None None] MFA Rules not processed for user `a1e1c24849d7480b8ac804a7e3d2ab5a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:54:41.386299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4ac22e36-cbf3-46d2-be50-b88f4cf9c87a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:54:41.393826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1149/2315] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:54:41 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:54:42.993960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=28 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:54:51.378680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:03.962052 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494ed442-e6e2-4ee6-a196-544b55d5a748 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:03.962572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494ed442-e6e2-4ee6-a196-544b55d5a748 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:03.962713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494ed442-e6e2-4ee6-a196-544b55d5a748 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:03.983593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-494ed442-e6e2-4ee6-a196-544b55d5a748 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:03.984869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-494ed442-e6e2-4ee6-a196-544b55d5a748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:03.991979 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1167/2316] 10.222.0.40 () {58 vars in 967 bytes} [Fri Jul 29 04:55:03 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:55:04.047938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac22e36-cbf3-46d2-be50-b88f4cf9c87a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:04.059903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac22e36-cbf3-46d2-be50-b88f4cf9c87a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:04.069024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0d7f6a-8898-4105-a236-fe6938b70b14 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:04.069653 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0d7f6a-8898-4105-a236-fe6938b70b14 service swift] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:04.069943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0d7f6a-8898-4105-a236-fe6938b70b14 service swift] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:04.070223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0d7f6a-8898-4105-a236-fe6938b70b14 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:04.079607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0d7f6a-8898-4105-a236-fe6938b70b14 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:04.081236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0d7f6a-8898-4105-a236-fe6938b70b14 service swift] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:04.089287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1150/2317] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:55:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:55:07.405390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494ed442-e6e2-4ee6-a196-544b55d5a748 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:07.422385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494ed442-e6e2-4ee6-a196-544b55d5a748 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:07.431268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d370044-7660-4ab0-8de5-f9b998255d27 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:07.431929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d370044-7660-4ab0-8de5-f9b998255d27 service placement] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:07.432229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d370044-7660-4ab0-8de5-f9b998255d27 service placement] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:07.432509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d370044-7660-4ab0-8de5-f9b998255d27 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:07.434708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3d370044-7660-4ab0-8de5-f9b998255d27 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:07.454244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d370044-7660-4ab0-8de5-f9b998255d27 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:07.456583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d370044-7660-4ab0-8de5-f9b998255d27 service placement] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:07.472644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1168/2318] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:55:07 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:55:10.751702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0d7f6a-8898-4105-a236-fe6938b70b14 service swift] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:10.772132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0d7f6a-8898-4105-a236-fe6938b70b14 service swift] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:10.782531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60546174-1e7c-4864-8732-2fee29de32fb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:10.783876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60546174-1e7c-4864-8732-2fee29de32fb service placement] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:10.784171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60546174-1e7c-4864-8732-2fee29de32fb service placement] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:10.784471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60546174-1e7c-4864-8732-2fee29de32fb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:10.786455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-60546174-1e7c-4864-8732-2fee29de32fb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:10.801798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60546174-1e7c-4864-8732-2fee29de32fb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:10.803903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60546174-1e7c-4864-8732-2fee29de32fb service placement] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:10.821018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1151/2319] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:55:10 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:55:10.847176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d370044-7660-4ab0-8de5-f9b998255d27 service placement] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:10.928771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d370044-7660-4ab0-8de5-f9b998255d27 service placement] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:10.933459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:10.934137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:10.934425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:10.934706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] PATH_INFO: `/v3/users/11f44b37b46b4793944285bee7b8427e` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:10.936510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] RBAC: Authorizing `identity:delete_user(user_id=11f44b37b46b4793944285bee7b8427e)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:10.938460 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:10.986254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11f44b37b46b4793944285bee7b8427e'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:10.986903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11f44b37b46b4793944285bee7b8427e'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:10.987521 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11f44b37b46b4793944285bee7b8427e'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:10.999981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11f44b37b46b4793944285bee7b8427e'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.006697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11f44b37b46b4793944285bee7b8427e'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.020259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1169/2320] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:55:10 2022] DELETE /identity/v3/users/11f44b37b46b4793944285bee7b8427e => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:11.195412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60546174-1e7c-4864-8732-2fee29de32fb service placement] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:11.209569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60546174-1e7c-4864-8732-2fee29de32fb service placement] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:11.273422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:11.274001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:11.274288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:11.274567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] PATH_INFO: `/v3/projects/21c0f0b9063948a9a0183b528bc21130` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:11.275798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=21c0f0b9063948a9a0183b528bc21130)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:11.277795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:11.332007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '21c0f0b9063948a9a0183b528bc21130', 'request_id': 'req-6e25d39c-092d-4884-80e9-2054c5c31d7e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.339399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '21c0f0b9063948a9a0183b528bc21130', 'request_id': 'req-6e25d39c-092d-4884-80e9-2054c5c31d7e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.346930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '21c0f0b9063948a9a0183b528bc21130', 'request_id': 'req-6e25d39c-092d-4884-80e9-2054c5c31d7e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.359431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] The token cache is being invalidate because project 21c0f0b9063948a9a0183b528bc21130 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:55:11.359727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.360441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.362044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1152/2321] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:55:11 2022] DELETE /identity/v3/projects/21c0f0b9063948a9a0183b528bc21130 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:11.367266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:11.368772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:11.382449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02744dd-c282-4408-809b-033cb2d0a864 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:11.443975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:11.444810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:11.445219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:11.445635 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] PATH_INFO: `/v3/users/d0548a60546d4ae581a9dd32320a3a57` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:11.446916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] RBAC: Authorizing `identity:delete_user(user_id=d0548a60546d4ae581a9dd32320a3a57)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:11.449025 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:11.450765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:11.506999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd0548a60546d4ae581a9dd32320a3a57'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.507795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd0548a60546d4ae581a9dd32320a3a57'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.508520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd0548a60546d4ae581a9dd32320a3a57'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.525281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd0548a60546d4ae581a9dd32320a3a57'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.533327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-39d6df46-2fff-4e5f-a11a-d88d66f9c93d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd0548a60546d4ae581a9dd32320a3a57'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.549429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1170/2322] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:55:11 2022] DELETE /identity/v3/users/d0548a60546d4ae581a9dd32320a3a57 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:55:11.726485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:11.728351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:11.760659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e25d39c-092d-4884-80e9-2054c5c31d7e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:11.829924 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:11.830682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:11.831050 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:11.831541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] PATH_INFO: `/v3/projects/824084791d65402f8a85aba10357e892` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:11.833015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=824084791d65402f8a85aba10357e892)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:11.835164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:11.843461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:11.901816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '824084791d65402f8a85aba10357e892', 'request_id': 'req-71527fe4-4a77-4f12-8afe-8be81782d7a3'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.909761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '824084791d65402f8a85aba10357e892', 'request_id': 'req-71527fe4-4a77-4f12-8afe-8be81782d7a3'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.915778 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '824084791d65402f8a85aba10357e892', 'request_id': 'req-71527fe4-4a77-4f12-8afe-8be81782d7a3'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.927305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] The token cache is being invalidate because project 824084791d65402f8a85aba10357e892 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:55:11.927598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.928254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:11.929763 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1153/2323] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:55:11 2022] DELETE /identity/v3/projects/824084791d65402f8a85aba10357e892 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:16.892315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7768119c-ccba-4de7-b019-a0b8fff7e37d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:16.892901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7768119c-ccba-4de7-b019-a0b8fff7e37d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:16.892901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7768119c-ccba-4de7-b019-a0b8fff7e37d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:16.923410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7768119c-ccba-4de7-b019-a0b8fff7e37d None None] MFA Rules not processed for user `8b7751b93f88424abebe6859f46f8351`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:16.924694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7768119c-ccba-4de7-b019-a0b8fff7e37d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:16.997226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1171/2324] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:16 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:21.924271 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=22,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:26.637216 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:26.640369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:26.878966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71527fe4-4a77-4f12-8afe-8be81782d7a3 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:26.885735 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39805ebf-f5f6-4e48-8bd5-2ae9bf21f7d0 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:26.885735 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39805ebf-f5f6-4e48-8bd5-2ae9bf21f7d0 service placement] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:26.886141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39805ebf-f5f6-4e48-8bd5-2ae9bf21f7d0 service placement] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:26.886670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39805ebf-f5f6-4e48-8bd5-2ae9bf21f7d0 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:26.888931 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-39805ebf-f5f6-4e48-8bd5-2ae9bf21f7d0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:26.904673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39805ebf-f5f6-4e48-8bd5-2ae9bf21f7d0 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:26.907248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39805ebf-f5f6-4e48-8bd5-2ae9bf21f7d0 service placement] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:26.917806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1154/2325] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:55:26 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:55:26.994792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=63,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:27.785077 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7768119c-ccba-4de7-b019-a0b8fff7e37d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:27.786622 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7768119c-ccba-4de7-b019-a0b8fff7e37d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:27.844668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7768119c-ccba-4de7-b019-a0b8fff7e37d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:27.848847 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2ebe9-0bf3-4361-9a9e-75d433d14cb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:27.849565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2ebe9-0bf3-4361-9a9e-75d433d14cb3 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:27.849914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2ebe9-0bf3-4361-9a9e-75d433d14cb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:27.850225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2ebe9-0bf3-4361-9a9e-75d433d14cb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:27.852242 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e2c2ebe9-0bf3-4361-9a9e-75d433d14cb3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:27.946981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c2ebe9-0bf3-4361-9a9e-75d433d14cb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:27.949091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c2ebe9-0bf3-4361-9a9e-75d433d14cb3 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:27.976539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1172/2326] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:55:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:55:32.407373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39805ebf-f5f6-4e48-8bd5-2ae9bf21f7d0 service placement] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:32.409105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-39805ebf-f5f6-4e48-8bd5-2ae9bf21f7d0 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:32.424256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39805ebf-f5f6-4e48-8bd5-2ae9bf21f7d0 service placement] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:32.429005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:32.429734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:32.430094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:32.430450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] PATH_INFO: `/v3/users/5f9d45265c6f47609f9f8abcf3526ff0` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:32.431712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f9d45265c6f47609f9f8abcf3526ff0)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:32.433804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:32.491901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f9d45265c6f47609f9f8abcf3526ff0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:32.501200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f9d45265c6f47609f9f8abcf3526ff0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:32.501669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f9d45265c6f47609f9f8abcf3526ff0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:32.502287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f9d45265c6f47609f9f8abcf3526ff0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:32.510901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d5199d73-cb07-4e0a-868f-dc738f427d41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f9d45265c6f47609f9f8abcf3526ff0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:32.530766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1155/2327] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:55:32 2022] DELETE /identity/v3/users/5f9d45265c6f47609f9f8abcf3526ff0 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:32.768345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2ebe9-0bf3-4361-9a9e-75d433d14cb3 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:32.791217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2ebe9-0bf3-4361-9a9e-75d433d14cb3 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:32.859137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:32.859938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:32.860330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:32.860753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] PATH_INFO: `/v3/projects/0f6c1ad3422c40d4af4ef5329ecd9d86` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:32.862071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f6c1ad3422c40d4af4ef5329ecd9d86)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:32.864268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:32.896023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:32.926394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f6c1ad3422c40d4af4ef5329ecd9d86', 'request_id': 'req-73bf7023-95d0-485f-89da-817c5f753610'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:32.933070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f6c1ad3422c40d4af4ef5329ecd9d86', 'request_id': 'req-73bf7023-95d0-485f-89da-817c5f753610'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:32.942011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f6c1ad3422c40d4af4ef5329ecd9d86', 'request_id': 'req-73bf7023-95d0-485f-89da-817c5f753610'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:32.955098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] The token cache is being invalidate because project 0f6c1ad3422c40d4af4ef5329ecd9d86 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:55:32.955402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:32.956431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-73bf7023-95d0-485f-89da-817c5f753610 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:32.958199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1173/2328] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:55:32 2022] DELETE /identity/v3/projects/0f6c1ad3422c40d4af4ef5329ecd9d86 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:32.962987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54cfe85-fe1c-49e8-ae1f-4cc815b985b6 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:32.963287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54cfe85-fe1c-49e8-ae1f-4cc815b985b6 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:32.963576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54cfe85-fe1c-49e8-ae1f-4cc815b985b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:32.969160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:33.040042 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f54cfe85-fe1c-49e8-ae1f-4cc815b985b6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:33.042014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f54cfe85-fe1c-49e8-ae1f-4cc815b985b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:33.113480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1156/2329] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:33.122252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d3c84b-0c83-46bf-b64f-b054f287ff0f None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:33.122558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d3c84b-0c83-46bf-b64f-b054f287ff0f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:33.122830 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d3c84b-0c83-46bf-b64f-b054f287ff0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:33.144510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-29d3c84b-0c83-46bf-b64f-b054f287ff0f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:33.145785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-29d3c84b-0c83-46bf-b64f-b054f287ff0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:33.153869 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1174/2330] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:55:33.158250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54cfe85-fe1c-49e8-ae1f-4cc815b985b6 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:33.169946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54cfe85-fe1c-49e8-ae1f-4cc815b985b6 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:33.175455 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae650f5-9ed2-498a-9b38-fa8d60c079e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:33.176094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae650f5-9ed2-498a-9b38-fa8d60c079e1 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:33.176384 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae650f5-9ed2-498a-9b38-fa8d60c079e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:33.176699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae650f5-9ed2-498a-9b38-fa8d60c079e1 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:33.178468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae650f5-9ed2-498a-9b38-fa8d60c079e1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:55:33.178468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae650f5-9ed2-498a-9b38-fa8d60c079e1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:33.179871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae650f5-9ed2-498a-9b38-fa8d60c079e1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:33.192924 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1157/2331] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:55:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:55:33.197300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d3c84b-0c83-46bf-b64f-b054f287ff0f None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:33.207548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d3c84b-0c83-46bf-b64f-b054f287ff0f None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:33.212374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00fb9cf-8187-46de-934a-b233faffbb33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:33.213057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00fb9cf-8187-46de-934a-b233faffbb33 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:33.213447 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00fb9cf-8187-46de-934a-b233faffbb33 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:33.213675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00fb9cf-8187-46de-934a-b233faffbb33 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:33.214595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00fb9cf-8187-46de-934a-b233faffbb33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:33.216757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00fb9cf-8187-46de-934a-b233faffbb33 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:33.244983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1175/2332] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:55:33 2022] POST /identity/v3/projects => generated 367 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:55:33.249718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae650f5-9ed2-498a-9b38-fa8d60c079e1 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:33.259653 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae650f5-9ed2-498a-9b38-fa8d60c079e1 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:33.319566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd10b1a-cb6d-44c0-a4f9-21edd5e11758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:33.320197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd10b1a-cb6d-44c0-a4f9-21edd5e11758 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:33.320532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd10b1a-cb6d-44c0-a4f9-21edd5e11758 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:33.320839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd10b1a-cb6d-44c0-a4f9-21edd5e11758 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:33.321534 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd10b1a-cb6d-44c0-a4f9-21edd5e11758 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:33.323539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd10b1a-cb6d-44c0-a4f9-21edd5e11758 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:33.348589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1158/2333] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:55:33 2022] POST /identity/v3/users => generated 284 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:55:33.352844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00fb9cf-8187-46de-934a-b233faffbb33 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:33.363785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00fb9cf-8187-46de-934a-b233faffbb33 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:33.369639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9473d7ce-35dc-48dd-9d31-04e1a2221647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:33.370621 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9473d7ce-35dc-48dd-9d31-04e1a2221647 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:33.370912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9473d7ce-35dc-48dd-9d31-04e1a2221647 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:33.371190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9473d7ce-35dc-48dd-9d31-04e1a2221647 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:33.371898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9473d7ce-35dc-48dd-9d31-04e1a2221647 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:33.373690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9473d7ce-35dc-48dd-9d31-04e1a2221647 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:33.384649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1176/2334] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:55:33 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:55:33.388729 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd10b1a-cb6d-44c0-a4f9-21edd5e11758 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:33.399250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd10b1a-cb6d-44c0-a4f9-21edd5e11758 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:33.404221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa241334-a719-4e4a-a157-cc0f1d75f3c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:33.406240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa241334-a719-4e4a-a157-cc0f1d75f3c2 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:33.406521 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa241334-a719-4e4a-a157-cc0f1d75f3c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:33.406801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa241334-a719-4e4a-a157-cc0f1d75f3c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:33.407539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa241334-a719-4e4a-a157-cc0f1d75f3c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:33.409180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa241334-a719-4e4a-a157-cc0f1d75f3c2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:33.424594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1159/2335] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:55:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:55:33.425745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9473d7ce-35dc-48dd-9d31-04e1a2221647 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:33.436262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9473d7ce-35dc-48dd-9d31-04e1a2221647 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:33.441730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679a0453-3bb1-43d8-9a72-7bbb8342777e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:33.442415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a0453-3bb1-43d8-9a72-7bbb8342777e admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:33.442704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a0453-3bb1-43d8-9a72-7bbb8342777e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:33.443038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a0453-3bb1-43d8-9a72-7bbb8342777e admin admin] PATH_INFO: `/v3/projects/acfceb52b610450a9b60e74a1932d9e4/users/c02475bd1f49459a925b1550c7f25dda/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:33.459718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679a0453-3bb1-43d8-9a72-7bbb8342777e admin admin] RBAC: Authorizing `identity:create_grant(project_id=acfceb52b610450a9b60e74a1932d9e4, user_id=c02475bd1f49459a925b1550c7f25dda, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:33.462437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679a0453-3bb1-43d8-9a72-7bbb8342777e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:33.476205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1177/2336] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:55:33 2022] PUT /identity/v3/projects/acfceb52b610450a9b60e74a1932d9e4/users/c02475bd1f49459a925b1550c7f25dda/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:33.486817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079e33ac-f79b-4edd-9926-55030a0b63e3 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:33.487142 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079e33ac-f79b-4edd-9926-55030a0b63e3 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:33.487426 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079e33ac-f79b-4edd-9926-55030a0b63e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:33.519097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-079e33ac-f79b-4edd-9926-55030a0b63e3 None None] MFA Rules not processed for user `c02475bd1f49459a925b1550c7f25dda`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:33.520730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-079e33ac-f79b-4edd-9926-55030a0b63e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:33.556898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:33.597659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1160/2337] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:33 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:33.609268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679a0453-3bb1-43d8-9a72-7bbb8342777e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:33.611194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-679a0453-3bb1-43d8-9a72-7bbb8342777e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:33.727873 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679a0453-3bb1-43d8-9a72-7bbb8342777e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:33.733611 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb105309-9635-48d1-bd94-3cc85b7a7d78 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:33.734235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb105309-9635-48d1-bd94-3cc85b7a7d78 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:33.734521 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb105309-9635-48d1-bd94-3cc85b7a7d78 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:33.734801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb105309-9635-48d1-bd94-3cc85b7a7d78 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:33.742638 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb105309-9635-48d1-bd94-3cc85b7a7d78 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:33.744213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb105309-9635-48d1-bd94-3cc85b7a7d78 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:33.753059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1178/2338] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:55:33 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:55:36.422357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079e33ac-f79b-4edd-9926-55030a0b63e3 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:36.482266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079e33ac-f79b-4edd-9926-55030a0b63e3 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:36.487638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:36.488283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:36.488570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:36.488897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] PATH_INFO: `/v3/users/c02475bd1f49459a925b1550c7f25dda` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:36.490068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c02475bd1f49459a925b1550c7f25dda)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:36.492001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:36.537898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c02475bd1f49459a925b1550c7f25dda'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:36.545859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c02475bd1f49459a925b1550c7f25dda'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:36.546489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c02475bd1f49459a925b1550c7f25dda'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:36.547176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c02475bd1f49459a925b1550c7f25dda'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:36.554116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c02475bd1f49459a925b1550c7f25dda'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:36.570823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1161/2339] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:55:36 2022] DELETE /identity/v3/users/c02475bd1f49459a925b1550c7f25dda => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:36.575408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccab36bd-8062-4fab-b714-1314945c11be None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:36.575890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccab36bd-8062-4fab-b714-1314945c11be None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:36.576008 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccab36bd-8062-4fab-b714-1314945c11be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:36.589650 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:DELETE=6,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:36.596167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-ccab36bd-8062-4fab-b714-1314945c11be None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:36.597892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ccab36bd-8062-4fab-b714-1314945c11be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:36.658004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1179/2340] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:36.668229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:36.669751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:36.722499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:36.751481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726fdfe9-b9dd-4427-b179-c4ecda2f5d3d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:36.755661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1542c1ff-5a80-4494-aa86-bc17cb27a915 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:36.756287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1542c1ff-5a80-4494-aa86-bc17cb27a915 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:36.756575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1542c1ff-5a80-4494-aa86-bc17cb27a915 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:36.756896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1542c1ff-5a80-4494-aa86-bc17cb27a915 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:36.766527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1542c1ff-5a80-4494-aa86-bc17cb27a915 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:36.767969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1542c1ff-5a80-4494-aa86-bc17cb27a915 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:36.777447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1162/2341] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:55:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:55:37.049958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccab36bd-8062-4fab-b714-1314945c11be None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:37.051424 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ccab36bd-8062-4fab-b714-1314945c11be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:37.064309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccab36bd-8062-4fab-b714-1314945c11be None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:37.069339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:37.070014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.070313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.070593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] PATH_INFO: `/v3/projects/acfceb52b610450a9b60e74a1932d9e4` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.072932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] RBAC: Authorizing `identity:delete_project(project_id=acfceb52b610450a9b60e74a1932d9e4)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:37.074965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:37.121964 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'acfceb52b610450a9b60e74a1932d9e4', 'request_id': 'req-cce6c280-c365-4fb5-a462-b724d6d4586b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:37.129061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'acfceb52b610450a9b60e74a1932d9e4', 'request_id': 'req-cce6c280-c365-4fb5-a462-b724d6d4586b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:37.136276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'acfceb52b610450a9b60e74a1932d9e4', 'request_id': 'req-cce6c280-c365-4fb5-a462-b724d6d4586b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:37.147793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] The token cache is being invalidate because project acfceb52b610450a9b60e74a1932d9e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:55:37.148090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:37.148761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-cce6c280-c365-4fb5-a462-b724d6d4586b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:37.150278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1180/2342] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:55:37 2022] DELETE /identity/v3/projects/acfceb52b610450a9b60e74a1932d9e4 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:37.155461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f043206-a09f-49c5-ac6c-e82bbec53960 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.155757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f043206-a09f-49c5-ac6c-e82bbec53960 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.156067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f043206-a09f-49c5-ac6c-e82bbec53960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.175555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-1f043206-a09f-49c5-ac6c-e82bbec53960 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:37.176742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-1f043206-a09f-49c5-ac6c-e82bbec53960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:37.241109 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1163/2343] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:37.249400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdafece9-9b18-45ca-abd1-c5f93cdc69ca None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.249775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdafece9-9b18-45ca-abd1-c5f93cdc69ca None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.250132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdafece9-9b18-45ca-abd1-c5f93cdc69ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.270575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-cdafece9-9b18-45ca-abd1-c5f93cdc69ca None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:37.271901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-cdafece9-9b18-45ca-abd1-c5f93cdc69ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:37.280186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1181/2344] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:37.284500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f043206-a09f-49c5-ac6c-e82bbec53960 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:37.294497 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f043206-a09f-49c5-ac6c-e82bbec53960 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:37.299345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6433a542-42db-4bae-ae4a-6740968abf85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:37.299969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6433a542-42db-4bae-ae4a-6740968abf85 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.300259 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6433a542-42db-4bae-ae4a-6740968abf85 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.300545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6433a542-42db-4bae-ae4a-6740968abf85 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.301286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6433a542-42db-4bae-ae4a-6740968abf85 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:55:37.301601 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6433a542-42db-4bae-ae4a-6740968abf85 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:37.303211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6433a542-42db-4bae-ae4a-6740968abf85 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:37.315764 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1164/2345] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:55:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:55:37.320144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdafece9-9b18-45ca-abd1-c5f93cdc69ca None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:37.330508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdafece9-9b18-45ca-abd1-c5f93cdc69ca None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:37.335222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8a7107-d4cf-44dc-b9a2-46ab4ab077f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:37.335837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8a7107-d4cf-44dc-b9a2-46ab4ab077f3 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.336124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8a7107-d4cf-44dc-b9a2-46ab4ab077f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.336403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8a7107-d4cf-44dc-b9a2-46ab4ab077f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.337126 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8a7107-d4cf-44dc-b9a2-46ab4ab077f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:37.339045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8a7107-d4cf-44dc-b9a2-46ab4ab077f3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:37.366757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1182/2346] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:55:37 2022] POST /identity/v3/projects => generated 367 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:55:37.370912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6433a542-42db-4bae-ae4a-6740968abf85 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:37.381739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6433a542-42db-4bae-ae4a-6740968abf85 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:37.438888 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e28a3a-8718-45e3-98ee-71a9d7557d39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:37.439500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e28a3a-8718-45e3-98ee-71a9d7557d39 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.439715 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e28a3a-8718-45e3-98ee-71a9d7557d39 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.440001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e28a3a-8718-45e3-98ee-71a9d7557d39 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.440733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e28a3a-8718-45e3-98ee-71a9d7557d39 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:37.442667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e28a3a-8718-45e3-98ee-71a9d7557d39 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:37.463563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1165/2347] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:55:37 2022] POST /identity/v3/users => generated 284 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:55:37.467921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8a7107-d4cf-44dc-b9a2-46ab4ab077f3 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:37.478325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8a7107-d4cf-44dc-b9a2-46ab4ab077f3 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:37.483570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76032a7-a373-4418-be02-3d6049f12f88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:37.484186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76032a7-a373-4418-be02-3d6049f12f88 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.484476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76032a7-a373-4418-be02-3d6049f12f88 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.484795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76032a7-a373-4418-be02-3d6049f12f88 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.485500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76032a7-a373-4418-be02-3d6049f12f88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:37.487105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76032a7-a373-4418-be02-3d6049f12f88 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:37.496922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1183/2348] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:55:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:55:37.501412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e28a3a-8718-45e3-98ee-71a9d7557d39 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:37.511181 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e28a3a-8718-45e3-98ee-71a9d7557d39 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:37.515986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684be814-e77a-467f-9a04-6c6dd7f8ab8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:37.516595 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684be814-e77a-467f-9a04-6c6dd7f8ab8d admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.516921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684be814-e77a-467f-9a04-6c6dd7f8ab8d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.517205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684be814-e77a-467f-9a04-6c6dd7f8ab8d admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.517915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684be814-e77a-467f-9a04-6c6dd7f8ab8d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:37.519524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684be814-e77a-467f-9a04-6c6dd7f8ab8d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:37.529994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1166/2349] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:55:37 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:55:37.533982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76032a7-a373-4418-be02-3d6049f12f88 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:37.544069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76032a7-a373-4418-be02-3d6049f12f88 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:37.549449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262daec1-2b2a-4fbc-bfac-b8a00ee284dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:37.550106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262daec1-2b2a-4fbc-bfac-b8a00ee284dd admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.550393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262daec1-2b2a-4fbc-bfac-b8a00ee284dd admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.550696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262daec1-2b2a-4fbc-bfac-b8a00ee284dd admin admin] PATH_INFO: `/v3/projects/538e1027ea0c4e72b0d4794689788cc4/users/9644f83564ec4124acbadc280644084c/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.565839 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262daec1-2b2a-4fbc-bfac-b8a00ee284dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=538e1027ea0c4e72b0d4794689788cc4, user_id=9644f83564ec4124acbadc280644084c, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:37.568479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262daec1-2b2a-4fbc-bfac-b8a00ee284dd admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:37.581196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1184/2350] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:55:37 2022] PUT /identity/v3/projects/538e1027ea0c4e72b0d4794689788cc4/users/9644f83564ec4124acbadc280644084c/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:37.592511 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dac7509-ce32-4ec4-9f53-9840fe86d25b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.592840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dac7509-ce32-4ec4-9f53-9840fe86d25b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.593129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dac7509-ce32-4ec4-9f53-9840fe86d25b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.625119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-9dac7509-ce32-4ec4-9f53-9840fe86d25b None None] MFA Rules not processed for user `9644f83564ec4124acbadc280644084c`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:37.626384 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9dac7509-ce32-4ec4-9f53-9840fe86d25b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:37.649877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:37.692486 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1167/2351] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:37 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:37.703275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262daec1-2b2a-4fbc-bfac-b8a00ee284dd admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:37.705065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-262daec1-2b2a-4fbc-bfac-b8a00ee284dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:37.736164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:37.764398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262daec1-2b2a-4fbc-bfac-b8a00ee284dd admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:37.768136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad706339-c33c-468e-add6-b71c8965ac1d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:37.768795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad706339-c33c-468e-add6-b71c8965ac1d service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:37.769089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad706339-c33c-468e-add6-b71c8965ac1d service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:37.769370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad706339-c33c-468e-add6-b71c8965ac1d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:37.777525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad706339-c33c-468e-add6-b71c8965ac1d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:37.778962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad706339-c33c-468e-add6-b71c8965ac1d service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:37.788218 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1185/2352] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:55:37 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:55:46.127254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dac7509-ce32-4ec4-9f53-9840fe86d25b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:46.190221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dac7509-ce32-4ec4-9f53-9840fe86d25b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:46.194654 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:46.195288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:46.195571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:46.195848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] PATH_INFO: `/v3/users/9644f83564ec4124acbadc280644084c` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:46.197072 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9644f83564ec4124acbadc280644084c)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:46.199014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:46.246037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9644f83564ec4124acbadc280644084c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:46.256061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9644f83564ec4124acbadc280644084c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:46.256742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9644f83564ec4124acbadc280644084c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:46.257394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9644f83564ec4124acbadc280644084c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:46.265537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9644f83564ec4124acbadc280644084c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:46.286250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1168/2353] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:55:46 2022] DELETE /identity/v3/users/9644f83564ec4124acbadc280644084c => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:46.291014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3727c8-ae68-4c30-b542-d697458d8f17 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:46.291493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3727c8-ae68-4c30-b542-d697458d8f17 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:46.291597 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3727c8-ae68-4c30-b542-d697458d8f17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:46.310582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-8b3727c8-ae68-4c30-b542-d697458d8f17 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:46.312256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8b3727c8-ae68-4c30-b542-d697458d8f17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:46.371073 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1186/2354] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:46.382073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:46.383500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:46.430870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:46.444070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102165f-96dc-4211-99f7-b1339ff7d7e8 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:46.448246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2547704-4161-44fb-a812-d55724382e92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:46.448902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2547704-4161-44fb-a812-d55724382e92 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:46.449194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2547704-4161-44fb-a812-d55724382e92 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:46.449479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2547704-4161-44fb-a812-d55724382e92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:46.459089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2547704-4161-44fb-a812-d55724382e92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:46.460519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2547704-4161-44fb-a812-d55724382e92 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:46.470118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1169/2355] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:55:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:55:46.731171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3727c8-ae68-4c30-b542-d697458d8f17 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:46.733462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8b3727c8-ae68-4c30-b542-d697458d8f17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:46.746586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3727c8-ae68-4c30-b542-d697458d8f17 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:46.752065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:46.752746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:46.753035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:46.753308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] PATH_INFO: `/v3/projects/538e1027ea0c4e72b0d4794689788cc4` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:46.754476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] RBAC: Authorizing `identity:delete_project(project_id=538e1027ea0c4e72b0d4794689788cc4)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:46.756453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:46.803854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '538e1027ea0c4e72b0d4794689788cc4', 'request_id': 'req-83ef2114-11f4-485b-99fe-2683e6a9ce56'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:46.810584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '538e1027ea0c4e72b0d4794689788cc4', 'request_id': 'req-83ef2114-11f4-485b-99fe-2683e6a9ce56'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:46.817797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '538e1027ea0c4e72b0d4794689788cc4', 'request_id': 'req-83ef2114-11f4-485b-99fe-2683e6a9ce56'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:46.829419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] The token cache is being invalidate because project 538e1027ea0c4e72b0d4794689788cc4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:55:46.829713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:46.830345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-83ef2114-11f4-485b-99fe-2683e6a9ce56 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:55:46.831855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1187/2356] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:55:46 2022] DELETE /identity/v3/projects/538e1027ea0c4e72b0d4794689788cc4 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:46.836872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b96931-f4fd-4177-85dc-60e8b9f3abd6 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:46.837159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b96931-f4fd-4177-85dc-60e8b9f3abd6 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:46.837447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b96931-f4fd-4177-85dc-60e8b9f3abd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:46.856571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-41b96931-f4fd-4177-85dc-60e8b9f3abd6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:46.857796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-41b96931-f4fd-4177-85dc-60e8b9f3abd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:46.915760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1170/2357] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:46.923584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45041117-33b2-476e-afc6-3fca191bc0b2 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:46.923913 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45041117-33b2-476e-afc6-3fca191bc0b2 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:46.924198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45041117-33b2-476e-afc6-3fca191bc0b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:46.944555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-45041117-33b2-476e-afc6-3fca191bc0b2 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:46.945800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-45041117-33b2-476e-afc6-3fca191bc0b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:46.954475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1188/2358] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:46.958621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b96931-f4fd-4177-85dc-60e8b9f3abd6 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:46.968191 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b96931-f4fd-4177-85dc-60e8b9f3abd6 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:46.972737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d03a6cd-63f0-4980-ba41-aff313f2e369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:46.973341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d03a6cd-63f0-4980-ba41-aff313f2e369 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:46.973625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d03a6cd-63f0-4980-ba41-aff313f2e369 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:46.973903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d03a6cd-63f0-4980-ba41-aff313f2e369 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:46.974597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d03a6cd-63f0-4980-ba41-aff313f2e369 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:55:46.974902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d03a6cd-63f0-4980-ba41-aff313f2e369 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:46.976493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d03a6cd-63f0-4980-ba41-aff313f2e369 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:46.987825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1171/2359] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:55:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:55:46.991955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45041117-33b2-476e-afc6-3fca191bc0b2 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.001898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45041117-33b2-476e-afc6-3fca191bc0b2 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.006701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460a9eb4-2715-4bf5-9b83-1d7a55695d99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.007311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a9eb4-2715-4bf5-9b83-1d7a55695d99 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.007603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a9eb4-2715-4bf5-9b83-1d7a55695d99 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.007887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a9eb4-2715-4bf5-9b83-1d7a55695d99 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.008666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460a9eb4-2715-4bf5-9b83-1d7a55695d99 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.010573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460a9eb4-2715-4bf5-9b83-1d7a55695d99 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.037644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1189/2360] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:55:46 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:55:47.042001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d03a6cd-63f0-4980-ba41-aff313f2e369 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.051984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d03a6cd-63f0-4980-ba41-aff313f2e369 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.107370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f5ea0c-74e1-4295-aef6-7259128be821 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.108054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5ea0c-74e1-4295-aef6-7259128be821 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.108343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5ea0c-74e1-4295-aef6-7259128be821 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.108649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f5ea0c-74e1-4295-aef6-7259128be821 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.109333 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f5ea0c-74e1-4295-aef6-7259128be821 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.111254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f5ea0c-74e1-4295-aef6-7259128be821 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.132207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1172/2361] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:55:47 2022] POST /identity/v3/users => generated 288 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:55:47.136421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460a9eb4-2715-4bf5-9b83-1d7a55695d99 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.147423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460a9eb4-2715-4bf5-9b83-1d7a55695d99 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.152035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d230257-1ef6-423a-b695-0d62c5faa770 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.152679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d230257-1ef6-423a-b695-0d62c5faa770 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.152972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d230257-1ef6-423a-b695-0d62c5faa770 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.153254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d230257-1ef6-423a-b695-0d62c5faa770 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.154010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d230257-1ef6-423a-b695-0d62c5faa770 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.155611 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d230257-1ef6-423a-b695-0d62c5faa770 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.165687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1190/2362] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:55:47.170163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f5ea0c-74e1-4295-aef6-7259128be821 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.179883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f5ea0c-74e1-4295-aef6-7259128be821 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.184389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1abb1e-a4a4-4347-996f-69cce2bff4f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.185646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1abb1e-a4a4-4347-996f-69cce2bff4f9 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.185934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1abb1e-a4a4-4347-996f-69cce2bff4f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.186212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1abb1e-a4a4-4347-996f-69cce2bff4f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.186899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1abb1e-a4a4-4347-996f-69cce2bff4f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.188515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1abb1e-a4a4-4347-996f-69cce2bff4f9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.198912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1173/2363] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:55:47.202972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d230257-1ef6-423a-b695-0d62c5faa770 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.212880 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d230257-1ef6-423a-b695-0d62c5faa770 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.217746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74288a5e-ebd3-4932-979e-8f29fba4c56c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.218403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74288a5e-ebd3-4932-979e-8f29fba4c56c admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.218690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74288a5e-ebd3-4932-979e-8f29fba4c56c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.218970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74288a5e-ebd3-4932-979e-8f29fba4c56c admin admin] PATH_INFO: `/v3/projects/b5fb770fe8f240e8af720c5511081eec/users/467de9f233394198b9a107bce67a9825/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.229840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:47.235439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74288a5e-ebd3-4932-979e-8f29fba4c56c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5fb770fe8f240e8af720c5511081eec, user_id=467de9f233394198b9a107bce67a9825, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.239122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74288a5e-ebd3-4932-979e-8f29fba4c56c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.252631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1191/2364] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:55:47 2022] PUT /identity/v3/projects/b5fb770fe8f240e8af720c5511081eec/users/467de9f233394198b9a107bce67a9825/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:55:47.264467 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b20c96d-4bbe-4c91-b8fa-d42c5a083f83 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.264795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b20c96d-4bbe-4c91-b8fa-d42c5a083f83 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.265090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b20c96d-4bbe-4c91-b8fa-d42c5a083f83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.292100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6b20c96d-4bbe-4c91-b8fa-d42c5a083f83 None None] MFA Rules not processed for user `467de9f233394198b9a107bce67a9825`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:47.293332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6b20c96d-4bbe-4c91-b8fa-d42c5a083f83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:47.333873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:47.359263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1174/2365] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:47 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:55:47.363839 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74288a5e-ebd3-4932-979e-8f29fba4c56c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.427010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74288a5e-ebd3-4932-979e-8f29fba4c56c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.431631 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187cd49a-1026-4d32-aab3-a6338b0f9e2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.432250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187cd49a-1026-4d32-aab3-a6338b0f9e2e admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.432540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187cd49a-1026-4d32-aab3-a6338b0f9e2e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.432853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187cd49a-1026-4d32-aab3-a6338b0f9e2e admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.433533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187cd49a-1026-4d32-aab3-a6338b0f9e2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.436081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187cd49a-1026-4d32-aab3-a6338b0f9e2e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.462117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1192/2366] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:55:47 2022] POST /identity/v3/projects => generated 375 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:55:47.466146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b20c96d-4bbe-4c91-b8fa-d42c5a083f83 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.476819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b20c96d-4bbe-4c91-b8fa-d42c5a083f83 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.533714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95021a4-f021-41b3-bc8b-67e5c5e926f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.534335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95021a4-f021-41b3-bc8b-67e5c5e926f2 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.534618 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95021a4-f021-41b3-bc8b-67e5c5e926f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.534890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95021a4-f021-41b3-bc8b-67e5c5e926f2 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.535557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95021a4-f021-41b3-bc8b-67e5c5e926f2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.537508 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95021a4-f021-41b3-bc8b-67e5c5e926f2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.559275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1175/2367] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:55:47 2022] POST /identity/v3/users => generated 294 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:55:47.563475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187cd49a-1026-4d32-aab3-a6338b0f9e2e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.573840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187cd49a-1026-4d32-aab3-a6338b0f9e2e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.578601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db532fe1-2b6b-4e74-9a85-8e67fdd517d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.579213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db532fe1-2b6b-4e74-9a85-8e67fdd517d5 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.579501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db532fe1-2b6b-4e74-9a85-8e67fdd517d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.579779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db532fe1-2b6b-4e74-9a85-8e67fdd517d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.580472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db532fe1-2b6b-4e74-9a85-8e67fdd517d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.582135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db532fe1-2b6b-4e74-9a85-8e67fdd517d5 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.593300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1193/2368] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:55:47.597450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95021a4-f021-41b3-bc8b-67e5c5e926f2 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.608151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95021a4-f021-41b3-bc8b-67e5c5e926f2 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.615503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735e3299-59b0-41b9-9e4f-140e47f116b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.616397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735e3299-59b0-41b9-9e4f-140e47f116b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.616772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735e3299-59b0-41b9-9e4f-140e47f116b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.617466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735e3299-59b0-41b9-9e4f-140e47f116b0 admin admin] PATH_INFO: `/v3/projects/bd29aac84d4a47a382fe4d1c547aab4c/users/02dc98c9a0b749a5ba3db858588ad08d/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.634137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735e3299-59b0-41b9-9e4f-140e47f116b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd29aac84d4a47a382fe4d1c547aab4c, user_id=02dc98c9a0b749a5ba3db858588ad08d, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.636860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735e3299-59b0-41b9-9e4f-140e47f116b0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.650022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1176/2369] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:55:47 2022] PUT /identity/v3/projects/bd29aac84d4a47a382fe4d1c547aab4c/users/02dc98c9a0b749a5ba3db858588ad08d/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:47.654614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db532fe1-2b6b-4e74-9a85-8e67fdd517d5 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.664660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db532fe1-2b6b-4e74-9a85-8e67fdd517d5 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.719655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d87220-eb3f-432e-a2f6-6f43ce4ea091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.720196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d87220-eb3f-432e-a2f6-6f43ce4ea091 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.720481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d87220-eb3f-432e-a2f6-6f43ce4ea091 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.720885 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d87220-eb3f-432e-a2f6-6f43ce4ea091 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.721579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d87220-eb3f-432e-a2f6-6f43ce4ea091 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.723160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d87220-eb3f-432e-a2f6-6f43ce4ea091 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.733016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1194/2370] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:55:47.737685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735e3299-59b0-41b9-9e4f-140e47f116b0 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.749047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735e3299-59b0-41b9-9e4f-140e47f116b0 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.753612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0507cd-4975-43e5-ba41-214cfd989ec5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.754281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0507cd-4975-43e5-ba41-214cfd989ec5 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.754565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0507cd-4975-43e5-ba41-214cfd989ec5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.754846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff0507cd-4975-43e5-ba41-214cfd989ec5 admin admin] PATH_INFO: `/v3/projects/bd29aac84d4a47a382fe4d1c547aab4c/users/02dc98c9a0b749a5ba3db858588ad08d/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.756810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0507cd-4975-43e5-ba41-214cfd989ec5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd29aac84d4a47a382fe4d1c547aab4c, user_id=02dc98c9a0b749a5ba3db858588ad08d, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:47.759430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff0507cd-4975-43e5-ba41-214cfd989ec5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:47.770265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1177/2371] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:55:47 2022] PUT /identity/v3/projects/bd29aac84d4a47a382fe4d1c547aab4c/users/02dc98c9a0b749a5ba3db858588ad08d/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:55:47.780848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27c0b16-7608-4f16-a70c-30d310407a11 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.781149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27c0b16-7608-4f16-a70c-30d310407a11 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.781440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27c0b16-7608-4f16-a70c-30d310407a11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:47.809773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-e27c0b16-7608-4f16-a70c-30d310407a11 None None] MFA Rules not processed for user `02dc98c9a0b749a5ba3db858588ad08d`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:55:47.811615 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e27c0b16-7608-4f16-a70c-30d310407a11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:47.857253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:47.879855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1195/2372] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:55:47 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:55:47.891103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0507cd-4975-43e5-ba41-214cfd989ec5 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:47.892853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ff0507cd-4975-43e5-ba41-214cfd989ec5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:55:47.969582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff0507cd-4975-43e5-ba41-214cfd989ec5 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:47.973311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0fccc8-ca7d-466d-a0c1-416c34069c74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:47.973873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fccc8-ca7d-466d-a0c1-416c34069c74 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:47.974162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fccc8-ca7d-466d-a0c1-416c34069c74 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:47.974440 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0fccc8-ca7d-466d-a0c1-416c34069c74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:48.003766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:55:48.031867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0fccc8-ca7d-466d-a0c1-416c34069c74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:48.033339 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0fccc8-ca7d-466d-a0c1-416c34069c74 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:48.042433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1178/2373] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:55:47 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:55:54.720480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27c0b16-7608-4f16-a70c-30d310407a11 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:55:54.730731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27c0b16-7608-4f16-a70c-30d310407a11 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:55:54.735213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b11024-e858-486a-a312-2df14d7eb6db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:55:54.735823 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b11024-e858-486a-a312-2df14d7eb6db service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:55:54.736109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b11024-e858-486a-a312-2df14d7eb6db service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:55:54.736391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b11024-e858-486a-a312-2df14d7eb6db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:55:54.745204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b11024-e858-486a-a312-2df14d7eb6db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:55:54.746578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b11024-e858-486a-a312-2df14d7eb6db service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:55:54.756829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1196/2374] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:55:54 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:55:58.038188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=14 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:04.752526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=13 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:18.424088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f9253-733c-4baf-971b-49e9005e63b9 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:18.424714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f9253-733c-4baf-971b-49e9005e63b9 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:18.424714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89f9253-733c-4baf-971b-49e9005e63b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:18.444386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-b89f9253-733c-4baf-971b-49e9005e63b9 None None] MFA Rules not processed for user `572408fd157844dcae13000d8e9c4b7a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:18.445667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b89f9253-733c-4baf-971b-49e9005e63b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:18.500752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1179/2375] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:18 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:25.876490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b11024-e858-486a-a312-2df14d7eb6db service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:25.941785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b11024-e858-486a-a312-2df14d7eb6db service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:25.946996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:25.947298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:25.947586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:25.947861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] PATH_INFO: `/v3/users/467de9f233394198b9a107bce67a9825` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:25.949298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] RBAC: Authorizing `identity:delete_user(user_id=467de9f233394198b9a107bce67a9825)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:25.951372 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:26.011323 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '467de9f233394198b9a107bce67a9825'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:26.012129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '467de9f233394198b9a107bce67a9825'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:26.012905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '467de9f233394198b9a107bce67a9825'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:26.026439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '467de9f233394198b9a107bce67a9825'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:26.034522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '467de9f233394198b9a107bce67a9825'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:26.053289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1197/2376] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:56:25 2022] DELETE /identity/v3/users/467de9f233394198b9a107bce67a9825 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:26.058570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e1805f-5031-43e1-b695-0337166bbc06 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:26.059044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e1805f-5031-43e1-b695-0337166bbc06 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:26.059334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e1805f-5031-43e1-b695-0337166bbc06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:26.082753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-06e1805f-5031-43e1-b695-0337166bbc06 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:26.083996 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-06e1805f-5031-43e1-b695-0337166bbc06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:26.152847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1180/2377] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:26.162886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:26.164587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:26.228997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a2f61e-49b2-465c-a951-943ebb866075 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:26.232990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b336390-82b0-4778-826a-738e3c83959e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:26.234415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b336390-82b0-4778-826a-738e3c83959e service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:26.234847 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b336390-82b0-4778-826a-738e3c83959e service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:26.235289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b336390-82b0-4778-826a-738e3c83959e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:26.246931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b336390-82b0-4778-826a-738e3c83959e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:26.248517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b336390-82b0-4778-826a-738e3c83959e service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:26.261113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1198/2378] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:56:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:56:27.010038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e1805f-5031-43e1-b695-0337166bbc06 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:27.011474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-06e1805f-5031-43e1-b695-0337166bbc06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:27.024514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e1805f-5031-43e1-b695-0337166bbc06 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:27.029248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:27.029890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:27.030849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:27.031431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] PATH_INFO: `/v3/projects/b5fb770fe8f240e8af720c5511081eec` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:27.032714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5fb770fe8f240e8af720c5511081eec)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:27.035555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:27.089972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5fb770fe8f240e8af720c5511081eec', 'request_id': 'req-28f2cfe6-bbc0-42f2-a968-835e4db919d5'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.099126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5fb770fe8f240e8af720c5511081eec', 'request_id': 'req-28f2cfe6-bbc0-42f2-a968-835e4db919d5'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.105038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5fb770fe8f240e8af720c5511081eec', 'request_id': 'req-28f2cfe6-bbc0-42f2-a968-835e4db919d5'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.116167 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] The token cache is being invalidate because project b5fb770fe8f240e8af720c5511081eec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:56:27.116468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.117132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.118637 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1181/2379] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:56:27 2022] DELETE /identity/v3/projects/b5fb770fe8f240e8af720c5511081eec => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:27.122851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b336390-82b0-4778-826a-738e3c83959e service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:27.133836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b336390-82b0-4778-826a-738e3c83959e service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:27.152777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:27.201741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:27.202991 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:27.203275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:27.203559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] PATH_INFO: `/v3/users/02dc98c9a0b749a5ba3db858588ad08d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:27.204791 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] RBAC: Authorizing `identity:delete_user(user_id=02dc98c9a0b749a5ba3db858588ad08d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:27.206703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:27.256632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02dc98c9a0b749a5ba3db858588ad08d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.257463 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02dc98c9a0b749a5ba3db858588ad08d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.258079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02dc98c9a0b749a5ba3db858588ad08d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.271841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02dc98c9a0b749a5ba3db858588ad08d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.279429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ca73caae-f491-485f-ba03-9bbc6ade2454 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02dc98c9a0b749a5ba3db858588ad08d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.293596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1199/2380] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:56:27 2022] DELETE /identity/v3/users/02dc98c9a0b749a5ba3db858588ad08d => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:27.565084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:27.566900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:27.591067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f2cfe6-bbc0-42f2-a968-835e4db919d5 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:27.618501 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:27.654607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:27.655296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:27.655580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:27.655859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] PATH_INFO: `/v3/projects/bd29aac84d4a47a382fe4d1c547aab4c` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:27.657096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd29aac84d4a47a382fe4d1c547aab4c)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:27.659011 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:27.705130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd29aac84d4a47a382fe4d1c547aab4c', 'request_id': 'req-4b77de5c-b6a0-4e1c-9248-1195812a3775'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.712799 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd29aac84d4a47a382fe4d1c547aab4c', 'request_id': 'req-4b77de5c-b6a0-4e1c-9248-1195812a3775'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.718691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd29aac84d4a47a382fe4d1c547aab4c', 'request_id': 'req-4b77de5c-b6a0-4e1c-9248-1195812a3775'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.729838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] The token cache is being invalidate because project bd29aac84d4a47a382fe4d1c547aab4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:56:27.730135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.730772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-4b77de5c-b6a0-4e1c-9248-1195812a3775 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:27.732286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1182/2381] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:56:27 2022] DELETE /identity/v3/projects/bd29aac84d4a47a382fe4d1c547aab4c => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:27.737907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9eaa11-db8e-4a73-b2f8-d6b3650782b1 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:27.738200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9eaa11-db8e-4a73-b2f8-d6b3650782b1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:27.738480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9eaa11-db8e-4a73-b2f8-d6b3650782b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:27.758249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-3c9eaa11-db8e-4a73-b2f8-d6b3650782b1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:27.759450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3c9eaa11-db8e-4a73-b2f8-d6b3650782b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:27.817561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1200/2382] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:27.826660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335b9da-9747-4ccb-9c81-79a40ebef6f3 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:27.827556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335b9da-9747-4ccb-9c81-79a40ebef6f3 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:27.827911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0335b9da-9747-4ccb-9c81-79a40ebef6f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:27.847855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0335b9da-9747-4ccb-9c81-79a40ebef6f3 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:27.849204 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0335b9da-9747-4ccb-9c81-79a40ebef6f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:27.857231 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1183/2383] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:27.861701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9eaa11-db8e-4a73-b2f8-d6b3650782b1 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:27.871426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9eaa11-db8e-4a73-b2f8-d6b3650782b1 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:27.876235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be56773b-3acb-4710-af97-af36cba25cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:27.876874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56773b-3acb-4710-af97-af36cba25cf5 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:27.877162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56773b-3acb-4710-af97-af36cba25cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:27.877442 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56773b-3acb-4710-af97-af36cba25cf5 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:27.878172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be56773b-3acb-4710-af97-af36cba25cf5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:56:27.878482 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be56773b-3acb-4710-af97-af36cba25cf5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:27.880113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be56773b-3acb-4710-af97-af36cba25cf5 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:27.894178 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1201/2384] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:56:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:56:27.898480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0335b9da-9747-4ccb-9c81-79a40ebef6f3 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:27.909260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0335b9da-9747-4ccb-9c81-79a40ebef6f3 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:27.914202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36dd8fb-fbb8-4668-a1ef-eb0f3ad2e231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:27.914813 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36dd8fb-fbb8-4668-a1ef-eb0f3ad2e231 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:27.915100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36dd8fb-fbb8-4668-a1ef-eb0f3ad2e231 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:27.915380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36dd8fb-fbb8-4668-a1ef-eb0f3ad2e231 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:27.916055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36dd8fb-fbb8-4668-a1ef-eb0f3ad2e231 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:27.917990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36dd8fb-fbb8-4668-a1ef-eb0f3ad2e231 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:27.946783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1184/2385] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:56:27 2022] POST /identity/v3/projects => generated 387 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:56:27.950954 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be56773b-3acb-4710-af97-af36cba25cf5 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:27.961538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be56773b-3acb-4710-af97-af36cba25cf5 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:27.987233 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:28.020219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9101442-6b27-41ab-b16b-ad5c2fcc67dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.020875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9101442-6b27-41ab-b16b-ad5c2fcc67dd admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.021168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9101442-6b27-41ab-b16b-ad5c2fcc67dd admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.021449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9101442-6b27-41ab-b16b-ad5c2fcc67dd admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.022123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9101442-6b27-41ab-b16b-ad5c2fcc67dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.024054 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9101442-6b27-41ab-b16b-ad5c2fcc67dd admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.045873 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1202/2386] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:56:27 2022] POST /identity/v3/users => generated 294 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:56:28.050200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36dd8fb-fbb8-4668-a1ef-eb0f3ad2e231 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:28.060515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36dd8fb-fbb8-4668-a1ef-eb0f3ad2e231 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:28.065596 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2665f2f2-324b-431e-8fe2-b1b821178ee5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.066200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2665f2f2-324b-431e-8fe2-b1b821178ee5 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.066483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2665f2f2-324b-431e-8fe2-b1b821178ee5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.066759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2665f2f2-324b-431e-8fe2-b1b821178ee5 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.067448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2665f2f2-324b-431e-8fe2-b1b821178ee5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.069202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2665f2f2-324b-431e-8fe2-b1b821178ee5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.080187 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1185/2387] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:56:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:56:28.084160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9101442-6b27-41ab-b16b-ad5c2fcc67dd admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:28.094103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9101442-6b27-41ab-b16b-ad5c2fcc67dd admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:28.098746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7829b893-7755-4db7-a462-310adcb9efa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.099349 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7829b893-7755-4db7-a462-310adcb9efa1 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.099633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7829b893-7755-4db7-a462-310adcb9efa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.099908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7829b893-7755-4db7-a462-310adcb9efa1 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.100594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7829b893-7755-4db7-a462-310adcb9efa1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.102203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7829b893-7755-4db7-a462-310adcb9efa1 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.112713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1203/2388] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:56:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:56:28.116719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2665f2f2-324b-431e-8fe2-b1b821178ee5 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:28.127544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2665f2f2-324b-431e-8fe2-b1b821178ee5 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:28.132125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2819f3ed-c85b-44d6-86b0-72cceca6816e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.132817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819f3ed-c85b-44d6-86b0-72cceca6816e admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.133100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819f3ed-c85b-44d6-86b0-72cceca6816e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.133383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2819f3ed-c85b-44d6-86b0-72cceca6816e admin admin] PATH_INFO: `/v3/projects/f4ef4f0d92fa48b8ad187290a384d8af/users/46dbf0a125464dfbb449159b78de7057/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.148277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2819f3ed-c85b-44d6-86b0-72cceca6816e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f4ef4f0d92fa48b8ad187290a384d8af, user_id=46dbf0a125464dfbb449159b78de7057, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.150946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2819f3ed-c85b-44d6-86b0-72cceca6816e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.163644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1186/2389] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:56:28 2022] PUT /identity/v3/projects/f4ef4f0d92fa48b8ad187290a384d8af/users/46dbf0a125464dfbb449159b78de7057/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:28.174426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e742d7-51b3-434b-80d1-0fc1e7092027 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.174915 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e742d7-51b3-434b-80d1-0fc1e7092027 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.175010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e742d7-51b3-434b-80d1-0fc1e7092027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.203352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-87e742d7-51b3-434b-80d1-0fc1e7092027 None None] MFA Rules not processed for user `46dbf0a125464dfbb449159b78de7057`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:28.204575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-87e742d7-51b3-434b-80d1-0fc1e7092027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:28.271180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1204/2390] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:28.275629 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2819f3ed-c85b-44d6-86b0-72cceca6816e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:28.334146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:28.339598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2819f3ed-c85b-44d6-86b0-72cceca6816e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:28.345745 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e6cb2a-4630-4030-a0bf-6a7c34476b96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.346390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e6cb2a-4630-4030-a0bf-6a7c34476b96 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.346852 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e6cb2a-4630-4030-a0bf-6a7c34476b96 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.347299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e6cb2a-4630-4030-a0bf-6a7c34476b96 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.348176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e6cb2a-4630-4030-a0bf-6a7c34476b96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.350134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e6cb2a-4630-4030-a0bf-6a7c34476b96 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.381114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1187/2391] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:56:28 2022] POST /identity/v3/projects => generated 387 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:56:28.385111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e742d7-51b3-434b-80d1-0fc1e7092027 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:28.396655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e742d7-51b3-434b-80d1-0fc1e7092027 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:28.454546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebae6db1-f58e-4e71-acc8-a8b9f7b92680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.455908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebae6db1-f58e-4e71-acc8-a8b9f7b92680 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.456212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebae6db1-f58e-4e71-acc8-a8b9f7b92680 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.456520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebae6db1-f58e-4e71-acc8-a8b9f7b92680 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.457247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebae6db1-f58e-4e71-acc8-a8b9f7b92680 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.459286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebae6db1-f58e-4e71-acc8-a8b9f7b92680 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.481912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1205/2392] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:56:28 2022] POST /identity/v3/users => generated 300 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:56:28.486470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e6cb2a-4630-4030-a0bf-6a7c34476b96 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:28.496801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e6cb2a-4630-4030-a0bf-6a7c34476b96 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:28.502323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bd450f-60a6-4d53-80ed-c8bcea37f60b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.502930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd450f-60a6-4d53-80ed-c8bcea37f60b admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.503221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd450f-60a6-4d53-80ed-c8bcea37f60b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.503506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bd450f-60a6-4d53-80ed-c8bcea37f60b admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.504204 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bd450f-60a6-4d53-80ed-c8bcea37f60b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.505857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bd450f-60a6-4d53-80ed-c8bcea37f60b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.516454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1188/2393] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:56:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:56:28.520374 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebae6db1-f58e-4e71-acc8-a8b9f7b92680 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:28.530753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebae6db1-f58e-4e71-acc8-a8b9f7b92680 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:28.535417 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217e9749-00aa-46d6-a32e-7a3858ba3d91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.536171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217e9749-00aa-46d6-a32e-7a3858ba3d91 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.536466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217e9749-00aa-46d6-a32e-7a3858ba3d91 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.536786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217e9749-00aa-46d6-a32e-7a3858ba3d91 admin admin] PATH_INFO: `/v3/projects/2062a045385e4b1e83e04bd9d2f00b42/users/4dbbe123a4f14b37929cf6a9a5003158/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.552917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217e9749-00aa-46d6-a32e-7a3858ba3d91 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2062a045385e4b1e83e04bd9d2f00b42, user_id=4dbbe123a4f14b37929cf6a9a5003158, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.555598 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217e9749-00aa-46d6-a32e-7a3858ba3d91 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.570663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1206/2394] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:56:28 2022] PUT /identity/v3/projects/2062a045385e4b1e83e04bd9d2f00b42/users/4dbbe123a4f14b37929cf6a9a5003158/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:56:28.574536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bd450f-60a6-4d53-80ed-c8bcea37f60b admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:28.584734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bd450f-60a6-4d53-80ed-c8bcea37f60b admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:28.642836 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32112579-768b-4c3d-9eb6-38b6993c15a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.643559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32112579-768b-4c3d-9eb6-38b6993c15a6 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.643849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32112579-768b-4c3d-9eb6-38b6993c15a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.644125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32112579-768b-4c3d-9eb6-38b6993c15a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.644862 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32112579-768b-4c3d-9eb6-38b6993c15a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.646472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32112579-768b-4c3d-9eb6-38b6993c15a6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.661590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1189/2395] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:56:28 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:56:28.665456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217e9749-00aa-46d6-a32e-7a3858ba3d91 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:28.671650 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:28.678392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217e9749-00aa-46d6-a32e-7a3858ba3d91 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:28.683790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be394853-1c7c-4023-95b7-c784e38e397c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.684470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be394853-1c7c-4023-95b7-c784e38e397c admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.684794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be394853-1c7c-4023-95b7-c784e38e397c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.685084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be394853-1c7c-4023-95b7-c784e38e397c admin admin] PATH_INFO: `/v3/projects/2062a045385e4b1e83e04bd9d2f00b42/users/4dbbe123a4f14b37929cf6a9a5003158/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.688220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be394853-1c7c-4023-95b7-c784e38e397c admin admin] RBAC: Authorizing `identity:create_grant(project_id=2062a045385e4b1e83e04bd9d2f00b42, user_id=4dbbe123a4f14b37929cf6a9a5003158, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.690907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be394853-1c7c-4023-95b7-c784e38e397c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.702479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1207/2396] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:56:28 2022] PUT /identity/v3/projects/2062a045385e4b1e83e04bd9d2f00b42/users/4dbbe123a4f14b37929cf6a9a5003158/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:56:28.712786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1af8da-e3a9-4e63-bf02-9a385b2f850d None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.713164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1af8da-e3a9-4e63-bf02-9a385b2f850d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.713518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1af8da-e3a9-4e63-bf02-9a385b2f850d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.744150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-9d1af8da-e3a9-4e63-bf02-9a385b2f850d None None] MFA Rules not processed for user `4dbbe123a4f14b37929cf6a9a5003158`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:28.745535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9d1af8da-e3a9-4e63-bf02-9a385b2f850d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:28.815877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1190/2397] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:28 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:28.826528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be394853-1c7c-4023-95b7-c784e38e397c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:28.828502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-be394853-1c7c-4023-95b7-c784e38e397c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:28.893078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be394853-1c7c-4023-95b7-c784e38e397c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:28.896928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a169a4b-0a9f-4754-97a0-f4ef7d97b68c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:28.897572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a169a4b-0a9f-4754-97a0-f4ef7d97b68c service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:28.897861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a169a4b-0a9f-4754-97a0-f4ef7d97b68c service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:28.898140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a169a4b-0a9f-4754-97a0-f4ef7d97b68c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:28.907515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a169a4b-0a9f-4754-97a0-f4ef7d97b68c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:28.909013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a169a4b-0a9f-4754-97a0-f4ef7d97b68c service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:28.918637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1208/2398] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:56:28 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:56:29.054066 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1af8da-e3a9-4e63-bf02-9a385b2f850d None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:29.104408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:29.117510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1af8da-e3a9-4e63-bf02-9a385b2f850d None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:29.122202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:29.122875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:29.123164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:29.123446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] PATH_INFO: `/v3/users/46dbf0a125464dfbb449159b78de7057` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:29.125232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=46dbf0a125464dfbb449159b78de7057)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:29.127196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:29.175984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46dbf0a125464dfbb449159b78de7057'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.183065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46dbf0a125464dfbb449159b78de7057'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.183779 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46dbf0a125464dfbb449159b78de7057'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.184427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46dbf0a125464dfbb449159b78de7057'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.192026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46dbf0a125464dfbb449159b78de7057'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.209662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1191/2399] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:56:29 2022] DELETE /identity/v3/users/46dbf0a125464dfbb449159b78de7057 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:29.214276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be202b8-f956-4477-920b-f11e7250e09d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:29.214752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be202b8-f956-4477-920b-f11e7250e09d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:29.214851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be202b8-f956-4477-920b-f11e7250e09d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:29.234034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-5be202b8-f956-4477-920b-f11e7250e09d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:29.235273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5be202b8-f956-4477-920b-f11e7250e09d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:29.294999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1209/2400] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:29.305834 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:29.307603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:29.366156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f5a263-7fde-413d-9f6e-39ab34d8e6b0 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:29.369935 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2577895f-deb4-4ca6-972a-ce4241026c44 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:29.370496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2577895f-deb4-4ca6-972a-ce4241026c44 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:29.370795 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2577895f-deb4-4ca6-972a-ce4241026c44 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:29.371124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2577895f-deb4-4ca6-972a-ce4241026c44 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:29.382117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2577895f-deb4-4ca6-972a-ce4241026c44 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:29.383610 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2577895f-deb4-4ca6-972a-ce4241026c44 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:29.394531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1192/2401] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:56:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:56:29.744254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be202b8-f956-4477-920b-f11e7250e09d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:29.745735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5be202b8-f956-4477-920b-f11e7250e09d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:29.757983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be202b8-f956-4477-920b-f11e7250e09d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:29.763047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:29.763749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:29.763993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:29.764315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] PATH_INFO: `/v3/projects/f4ef4f0d92fa48b8ad187290a384d8af` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:29.765968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f4ef4f0d92fa48b8ad187290a384d8af)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:29.768818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:29.816702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f4ef4f0d92fa48b8ad187290a384d8af', 'request_id': 'req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.822818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f4ef4f0d92fa48b8ad187290a384d8af', 'request_id': 'req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.830711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f4ef4f0d92fa48b8ad187290a384d8af', 'request_id': 'req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.841019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] The token cache is being invalidate because project f4ef4f0d92fa48b8ad187290a384d8af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:56:29.841312 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.841963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.844249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1210/2402] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:56:29 2022] DELETE /identity/v3/projects/f4ef4f0d92fa48b8ad187290a384d8af => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:29.848273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2577895f-deb4-4ca6-972a-ce4241026c44 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:29.858213 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2577895f-deb4-4ca6-972a-ce4241026c44 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:29.915557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:29.916222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:29.916504 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:29.916805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] PATH_INFO: `/v3/users/4dbbe123a4f14b37929cf6a9a5003158` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:29.917961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] RBAC: Authorizing `identity:delete_user(user_id=4dbbe123a4f14b37929cf6a9a5003158)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:29.919886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:29.933139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:29.966715 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4dbbe123a4f14b37929cf6a9a5003158'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.973940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4dbbe123a4f14b37929cf6a9a5003158'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.974646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4dbbe123a4f14b37929cf6a9a5003158'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.975316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4dbbe123a4f14b37929cf6a9a5003158'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.982242 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f970991b-aa8e-43c8-ad41-81514836dd6d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4dbbe123a4f14b37929cf6a9a5003158'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:29.999688 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1193/2403] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:56:29 2022] DELETE /identity/v3/users/4dbbe123a4f14b37929cf6a9a5003158 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:30.269874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:30.271395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:30.284744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf09f5dd-4818-46f2-b696-8b5b60cd3c97 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:30.304474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:30.352408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:30.353118 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:30.353404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:30.353684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] PATH_INFO: `/v3/projects/2062a045385e4b1e83e04bd9d2f00b42` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:30.354899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=2062a045385e4b1e83e04bd9d2f00b42)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:30.356832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:30.403331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2062a045385e4b1e83e04bd9d2f00b42', 'request_id': 'req-90ab731f-1777-475e-9196-f8060f903c6f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:30.410017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2062a045385e4b1e83e04bd9d2f00b42', 'request_id': 'req-90ab731f-1777-475e-9196-f8060f903c6f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:30.417377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2062a045385e4b1e83e04bd9d2f00b42', 'request_id': 'req-90ab731f-1777-475e-9196-f8060f903c6f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:30.428751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] The token cache is being invalidate because project 2062a045385e4b1e83e04bd9d2f00b42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:56:30.429048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:30.429682 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90ab731f-1777-475e-9196-f8060f903c6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:30.431319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1211/2404] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:56:30 2022] DELETE /identity/v3/projects/2062a045385e4b1e83e04bd9d2f00b42 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:30.436794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22b462f-e154-4871-9984-b2ac8d717d06 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:30.437275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22b462f-e154-4871-9984-b2ac8d717d06 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:30.437453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22b462f-e154-4871-9984-b2ac8d717d06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:30.458762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e22b462f-e154-4871-9984-b2ac8d717d06 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:30.459973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e22b462f-e154-4871-9984-b2ac8d717d06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:30.520280 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1194/2405] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:30.584626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176c660f-b8b1-4f9a-ae14-25ce73adf7c7 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:30.584916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176c660f-b8b1-4f9a-ae14-25ce73adf7c7 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:30.585202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176c660f-b8b1-4f9a-ae14-25ce73adf7c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:30.604806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-176c660f-b8b1-4f9a-ae14-25ce73adf7c7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:30.606019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-176c660f-b8b1-4f9a-ae14-25ce73adf7c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:30.620801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1212/2406] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:30.625971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22b462f-e154-4871-9984-b2ac8d717d06 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:30.635758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22b462f-e154-4871-9984-b2ac8d717d06 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:30.640511 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de99da07-b570-4cf6-a35b-bac0d76b5d26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:30.641197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de99da07-b570-4cf6-a35b-bac0d76b5d26 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:30.641426 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de99da07-b570-4cf6-a35b-bac0d76b5d26 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:30.641698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de99da07-b570-4cf6-a35b-bac0d76b5d26 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:30.642394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de99da07-b570-4cf6-a35b-bac0d76b5d26 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:56:30.642699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de99da07-b570-4cf6-a35b-bac0d76b5d26 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:30.644288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de99da07-b570-4cf6-a35b-bac0d76b5d26 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:30.656995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1195/2407] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:56:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:56:30.666487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176c660f-b8b1-4f9a-ae14-25ce73adf7c7 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:30.676433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176c660f-b8b1-4f9a-ae14-25ce73adf7c7 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:30.681268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d602aaec-e893-4d1f-9e15-ab5ee0799ea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:30.681884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d602aaec-e893-4d1f-9e15-ab5ee0799ea2 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:30.682171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d602aaec-e893-4d1f-9e15-ab5ee0799ea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:30.682448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d602aaec-e893-4d1f-9e15-ab5ee0799ea2 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:30.683125 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d602aaec-e893-4d1f-9e15-ab5ee0799ea2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:30.685073 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d602aaec-e893-4d1f-9e15-ab5ee0799ea2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:30.719031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1213/2408] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:56:30 2022] POST /identity/v3/projects => generated 371 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:56:30.723216 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de99da07-b570-4cf6-a35b-bac0d76b5d26 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:30.734448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de99da07-b570-4cf6-a35b-bac0d76b5d26 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:30.802983 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea40ef72-a5f2-44d6-94a5-498470d1a4f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:30.803868 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea40ef72-a5f2-44d6-94a5-498470d1a4f1 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:30.804283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea40ef72-a5f2-44d6-94a5-498470d1a4f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:30.804746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea40ef72-a5f2-44d6-94a5-498470d1a4f1 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:30.805622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea40ef72-a5f2-44d6-94a5-498470d1a4f1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:30.808042 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea40ef72-a5f2-44d6-94a5-498470d1a4f1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:30.836370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1196/2409] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:56:30 2022] POST /identity/v3/users => generated 286 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:56:30.840670 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d602aaec-e893-4d1f-9e15-ab5ee0799ea2 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:30.852262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d602aaec-e893-4d1f-9e15-ab5ee0799ea2 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:30.857333 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c94530a-cc26-492f-ab08-d3148966700b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:30.858951 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c94530a-cc26-492f-ab08-d3148966700b admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:30.859612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c94530a-cc26-492f-ab08-d3148966700b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:30.859936 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c94530a-cc26-492f-ab08-d3148966700b admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:30.860661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c94530a-cc26-492f-ab08-d3148966700b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:30.863110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c94530a-cc26-492f-ab08-d3148966700b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:30.884653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1214/2410] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:56:30 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:56:30.888496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea40ef72-a5f2-44d6-94a5-498470d1a4f1 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:30.900422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea40ef72-a5f2-44d6-94a5-498470d1a4f1 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:30.905656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89113759-47a9-4919-af50-2e92da264ac6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:30.906371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89113759-47a9-4919-af50-2e92da264ac6 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:30.906730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89113759-47a9-4919-af50-2e92da264ac6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:30.907091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89113759-47a9-4919-af50-2e92da264ac6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:30.907979 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89113759-47a9-4919-af50-2e92da264ac6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:30.909777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89113759-47a9-4919-af50-2e92da264ac6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:30.912309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:30.926685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1197/2411] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:56:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:56:30.927925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c94530a-cc26-492f-ab08-d3148966700b admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:30.938836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c94530a-cc26-492f-ab08-d3148966700b admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:30.943909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ed5ed9-6e7f-4482-bd37-b9296805115c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:30.944585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ed5ed9-6e7f-4482-bd37-b9296805115c admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:30.944909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ed5ed9-6e7f-4482-bd37-b9296805115c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:30.945191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ed5ed9-6e7f-4482-bd37-b9296805115c admin admin] PATH_INFO: `/v3/projects/89dfac604f9740268351be452b8514fd/users/f03a509ef0b148ec9e93499b7b025665/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:30.963832 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ed5ed9-6e7f-4482-bd37-b9296805115c admin admin] RBAC: Authorizing `identity:create_grant(project_id=89dfac604f9740268351be452b8514fd, user_id=f03a509ef0b148ec9e93499b7b025665, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:30.966554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ed5ed9-6e7f-4482-bd37-b9296805115c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:30.980329 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1215/2412] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:56:30 2022] PUT /identity/v3/projects/89dfac604f9740268351be452b8514fd/users/f03a509ef0b148ec9e93499b7b025665/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:30.986260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848f3fd4-1369-4c6f-ba5a-87ae291e235f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:30.986555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848f3fd4-1369-4c6f-ba5a-87ae291e235f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:30.986836 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848f3fd4-1369-4c6f-ba5a-87ae291e235f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:31.007040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-848f3fd4-1369-4c6f-ba5a-87ae291e235f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:31.008273 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-848f3fd4-1369-4c6f-ba5a-87ae291e235f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:31.068925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1198/2413] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:31.078637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ed5ed9-6e7f-4482-bd37-b9296805115c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:31.080343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c7ed5ed9-6e7f-4482-bd37-b9296805115c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:31.127408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:31.140141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ed5ed9-6e7f-4482-bd37-b9296805115c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:31.144017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099946c7-7c4b-40ef-a187-5b1b813a0d33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:31.144672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099946c7-7c4b-40ef-a187-5b1b813a0d33 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:31.144966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099946c7-7c4b-40ef-a187-5b1b813a0d33 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:31.145246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099946c7-7c4b-40ef-a187-5b1b813a0d33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:31.154362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099946c7-7c4b-40ef-a187-5b1b813a0d33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:31.156326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099946c7-7c4b-40ef-a187-5b1b813a0d33 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:31.165933 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1216/2414] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:56:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:56:31.791666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07266711-3fff-44cd-ace1-4a860626fb4b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:31.792172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07266711-3fff-44cd-ace1-4a860626fb4b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:31.792294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07266711-3fff-44cd-ace1-4a860626fb4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:31.820689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-07266711-3fff-44cd-ace1-4a860626fb4b None None] MFA Rules not processed for user `f03a509ef0b148ec9e93499b7b025665`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:31.822245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-07266711-3fff-44cd-ace1-4a860626fb4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:31.897354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1199/2415] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:31 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:31.901817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099946c7-7c4b-40ef-a187-5b1b813a0d33 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:31.912117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099946c7-7c4b-40ef-a187-5b1b813a0d33 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:31.917014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0bb0ce-9921-430c-84c9-1173f492e0e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:31.917633 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0bb0ce-9921-430c-84c9-1173f492e0e0 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:31.917918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0bb0ce-9921-430c-84c9-1173f492e0e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:31.918194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0bb0ce-9921-430c-84c9-1173f492e0e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:31.918859 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0bb0ce-9921-430c-84c9-1173f492e0e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:31.921461 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0bb0ce-9921-430c-84c9-1173f492e0e0 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:31.956508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1217/2416] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:56:31 2022] POST /identity/v3/projects => generated 371 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:56:31.957965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07266711-3fff-44cd-ace1-4a860626fb4b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:31.969526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07266711-3fff-44cd-ace1-4a860626fb4b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:32.029365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4418d972-f488-4c2e-b127-a5c5c835a9f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:32.030644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4418d972-f488-4c2e-b127-a5c5c835a9f4 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:32.031004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4418d972-f488-4c2e-b127-a5c5c835a9f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:32.031521 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4418d972-f488-4c2e-b127-a5c5c835a9f4 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:32.032263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4418d972-f488-4c2e-b127-a5c5c835a9f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:32.034332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4418d972-f488-4c2e-b127-a5c5c835a9f4 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:32.060541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1200/2417] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:56:31 2022] POST /identity/v3/users => generated 292 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:56:32.062473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0bb0ce-9921-430c-84c9-1173f492e0e0 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:32.073036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0bb0ce-9921-430c-84c9-1173f492e0e0 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:32.078388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0925518-c84b-4ffe-a5b4-df347599117e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:32.079006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0925518-c84b-4ffe-a5b4-df347599117e admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:32.079307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0925518-c84b-4ffe-a5b4-df347599117e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:32.079629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0925518-c84b-4ffe-a5b4-df347599117e admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:32.080336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0925518-c84b-4ffe-a5b4-df347599117e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:32.081968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0925518-c84b-4ffe-a5b4-df347599117e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:32.094842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1218/2418] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:56:32 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:56:32.101896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4418d972-f488-4c2e-b127-a5c5c835a9f4 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:32.113404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4418d972-f488-4c2e-b127-a5c5c835a9f4 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:32.118148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c332421-377c-41fb-a53d-7fbff58482f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:32.118811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c332421-377c-41fb-a53d-7fbff58482f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:32.119102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c332421-377c-41fb-a53d-7fbff58482f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:32.119388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c332421-377c-41fb-a53d-7fbff58482f4 admin admin] PATH_INFO: `/v3/projects/28b82d50d9984e559adb18c825827fba/users/ae77413392dd43a8a070073d4367d4b1/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:32.134319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c332421-377c-41fb-a53d-7fbff58482f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28b82d50d9984e559adb18c825827fba, user_id=ae77413392dd43a8a070073d4367d4b1, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:32.137006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c332421-377c-41fb-a53d-7fbff58482f4 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:32.140029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:32.151126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1201/2419] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:56:32 2022] PUT /identity/v3/projects/28b82d50d9984e559adb18c825827fba/users/ae77413392dd43a8a070073d4367d4b1/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:32.155942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0925518-c84b-4ffe-a5b4-df347599117e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:32.166070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0925518-c84b-4ffe-a5b4-df347599117e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:32.222048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177376b5-5d61-4297-a609-65ac209dd5e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:32.222667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177376b5-5d61-4297-a609-65ac209dd5e1 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:32.222950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177376b5-5d61-4297-a609-65ac209dd5e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:32.223227 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177376b5-5d61-4297-a609-65ac209dd5e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:32.223911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177376b5-5d61-4297-a609-65ac209dd5e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:32.225525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177376b5-5d61-4297-a609-65ac209dd5e1 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:32.235271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1219/2420] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:56:32 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:56:32.239417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c332421-377c-41fb-a53d-7fbff58482f4 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:32.249751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c332421-377c-41fb-a53d-7fbff58482f4 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:32.254448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d46b9-90c2-4f3c-bf2e-be8bc1f62951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:32.255107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d46b9-90c2-4f3c-bf2e-be8bc1f62951 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:32.255394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d46b9-90c2-4f3c-bf2e-be8bc1f62951 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:32.255683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d46b9-90c2-4f3c-bf2e-be8bc1f62951 admin admin] PATH_INFO: `/v3/projects/28b82d50d9984e559adb18c825827fba/users/ae77413392dd43a8a070073d4367d4b1/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:32.257661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74d46b9-90c2-4f3c-bf2e-be8bc1f62951 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28b82d50d9984e559adb18c825827fba, user_id=ae77413392dd43a8a070073d4367d4b1, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:32.260308 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74d46b9-90c2-4f3c-bf2e-be8bc1f62951 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:32.271421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1202/2421] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:56:32 2022] PUT /identity/v3/projects/28b82d50d9984e559adb18c825827fba/users/ae77413392dd43a8a070073d4367d4b1/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:32.893914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0218fd06-3458-49c5-867b-54ae88f67c18 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:32.894214 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0218fd06-3458-49c5-867b-54ae88f67c18 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:32.894503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0218fd06-3458-49c5-867b-54ae88f67c18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:32.926627 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-0218fd06-3458-49c5-867b-54ae88f67c18 None None] MFA Rules not processed for user `ae77413392dd43a8a070073d4367d4b1`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:32.928794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0218fd06-3458-49c5-867b-54ae88f67c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:33.008930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1220/2422] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:32 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:33.022679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d46b9-90c2-4f3c-bf2e-be8bc1f62951 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:33.024537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e74d46b9-90c2-4f3c-bf2e-be8bc1f62951 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:33.085704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d46b9-90c2-4f3c-bf2e-be8bc1f62951 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:33.089437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1d70b2-c274-4a7a-8ffc-92b5ea9f899d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:33.090065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1d70b2-c274-4a7a-8ffc-92b5ea9f899d service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:33.090355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1d70b2-c274-4a7a-8ffc-92b5ea9f899d service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:33.090691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1d70b2-c274-4a7a-8ffc-92b5ea9f899d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:33.161048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1d70b2-c274-4a7a-8ffc-92b5ea9f899d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:33.162543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1d70b2-c274-4a7a-8ffc-92b5ea9f899d service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:33.175636 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1203/2423] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:56:33 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:56:34.567951 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0218fd06-3458-49c5-867b-54ae88f67c18 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:34.569958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0218fd06-3458-49c5-867b-54ae88f67c18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:34.583908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0218fd06-3458-49c5-867b-54ae88f67c18 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:34.588820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e8986e-81a7-4f59-8d11-895a31dfbb8e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:34.589470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e8986e-81a7-4f59-8d11-895a31dfbb8e service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:34.589749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e8986e-81a7-4f59-8d11-895a31dfbb8e service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:34.590028 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e8986e-81a7-4f59-8d11-895a31dfbb8e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:34.600820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e8986e-81a7-4f59-8d11-895a31dfbb8e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:34.602354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e8986e-81a7-4f59-8d11-895a31dfbb8e service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:34.612411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1221/2424] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:56:34 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:56:43.169671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:INSERT=2,keystone:SELECT=55 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:44.607888 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:47.565689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1d70b2-c274-4a7a-8ffc-92b5ea9f899d service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:47.634635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1d70b2-c274-4a7a-8ffc-92b5ea9f899d service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:47.639272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:47.639916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:47.640200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:47.640476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] PATH_INFO: `/v3/users/f03a509ef0b148ec9e93499b7b025665` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:47.641763 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f03a509ef0b148ec9e93499b7b025665)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:47.644345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:47.692578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f03a509ef0b148ec9e93499b7b025665'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:47.699696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f03a509ef0b148ec9e93499b7b025665'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:47.700344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f03a509ef0b148ec9e93499b7b025665'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:47.700981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f03a509ef0b148ec9e93499b7b025665'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:47.708587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f03a509ef0b148ec9e93499b7b025665'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:47.726091 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1204/2425] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:56:47 2022] DELETE /identity/v3/users/f03a509ef0b148ec9e93499b7b025665 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:47.903749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e8986e-81a7-4f59-8d11-895a31dfbb8e service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:47.917159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e8986e-81a7-4f59-8d11-895a31dfbb8e service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:47.980400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:47.981066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:47.981351 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:47.981628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] PATH_INFO: `/v3/projects/89dfac604f9740268351be452b8514fd` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:47.982771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] RBAC: Authorizing `identity:delete_project(project_id=89dfac604f9740268351be452b8514fd)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:47.984705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:48.030307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89dfac604f9740268351be452b8514fd', 'request_id': 'req-eb3559cd-fe52-48cf-a0a2-f933255faa1d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.036479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89dfac604f9740268351be452b8514fd', 'request_id': 'req-eb3559cd-fe52-48cf-a0a2-f933255faa1d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.044444 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89dfac604f9740268351be452b8514fd', 'request_id': 'req-eb3559cd-fe52-48cf-a0a2-f933255faa1d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.055198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] The token cache is being invalidate because project 89dfac604f9740268351be452b8514fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:56:48.055502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.056106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.057636 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1222/2426] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:56:47 2022] DELETE /identity/v3/projects/89dfac604f9740268351be452b8514fd => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:48.061633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:48.063122 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:48.076872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43111bb1-797a-447c-be62-2c8caba3b607 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:48.133088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:48.133722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.134010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.134291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] PATH_INFO: `/v3/users/ae77413392dd43a8a070073d4367d4b1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.135445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae77413392dd43a8a070073d4367d4b1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:48.137390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:48.159042 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:48.184514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae77413392dd43a8a070073d4367d4b1'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.192014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae77413392dd43a8a070073d4367d4b1'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.192690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae77413392dd43a8a070073d4367d4b1'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.193346 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae77413392dd43a8a070073d4367d4b1'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.200199 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-5e52781a-53a5-437a-87c2-ef2cec5241cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae77413392dd43a8a070073d4367d4b1'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.216865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1205/2427] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:56:48 2022] DELETE /identity/v3/users/ae77413392dd43a8a070073d4367d4b1 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:48.387013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:48.388696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:48.406206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3559cd-fe52-48cf-a0a2-f933255faa1d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:48.468775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:48.469423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.469712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.470694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] PATH_INFO: `/v3/projects/28b82d50d9984e559adb18c825827fba` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.471867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] RBAC: Authorizing `identity:delete_project(project_id=28b82d50d9984e559adb18c825827fba)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:48.473809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:48.506708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:48.520442 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28b82d50d9984e559adb18c825827fba', 'request_id': 'req-fdb087ae-316b-4ff0-b560-e512c1edb39c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.526540 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28b82d50d9984e559adb18c825827fba', 'request_id': 'req-fdb087ae-316b-4ff0-b560-e512c1edb39c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.534322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28b82d50d9984e559adb18c825827fba', 'request_id': 'req-fdb087ae-316b-4ff0-b560-e512c1edb39c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.544785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] The token cache is being invalidate because project 28b82d50d9984e559adb18c825827fba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:56:48.545082 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.545725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-fdb087ae-316b-4ff0-b560-e512c1edb39c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:56:48.547454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1223/2428] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:56:48 2022] DELETE /identity/v3/projects/28b82d50d9984e559adb18c825827fba => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:48.553766 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4753fc6-665c-40e3-a6e7-45bfe55f47ab None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.554092 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4753fc6-665c-40e3-a6e7-45bfe55f47ab None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.554365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4753fc6-665c-40e3-a6e7-45bfe55f47ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.575179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f4753fc6-665c-40e3-a6e7-45bfe55f47ab None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:48.577385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f4753fc6-665c-40e3-a6e7-45bfe55f47ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:48.641286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1206/2429] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:48.649722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e5ba8a-9659-473d-a9d3-f943a017a785 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.650088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e5ba8a-9659-473d-a9d3-f943a017a785 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.650446 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e5ba8a-9659-473d-a9d3-f943a017a785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.671793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-56e5ba8a-9659-473d-a9d3-f943a017a785 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:48.673143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-56e5ba8a-9659-473d-a9d3-f943a017a785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:48.681661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1224/2430] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:48.686093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4753fc6-665c-40e3-a6e7-45bfe55f47ab None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:48.695876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4753fc6-665c-40e3-a6e7-45bfe55f47ab None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:48.700359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6535e850-014b-4e44-8ede-9f1fbfc8156d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:48.701008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6535e850-014b-4e44-8ede-9f1fbfc8156d admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.701293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6535e850-014b-4e44-8ede-9f1fbfc8156d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.701569 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6535e850-014b-4e44-8ede-9f1fbfc8156d admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.702261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6535e850-014b-4e44-8ede-9f1fbfc8156d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:56:48.702565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6535e850-014b-4e44-8ede-9f1fbfc8156d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:48.704157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6535e850-014b-4e44-8ede-9f1fbfc8156d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:48.716433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1207/2431] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:56:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:56:48.721322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e5ba8a-9659-473d-a9d3-f943a017a785 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:48.730620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e5ba8a-9659-473d-a9d3-f943a017a785 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:48.735280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a33c304-c43c-481f-9f10-0bcea348a65e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:48.736019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a33c304-c43c-481f-9f10-0bcea348a65e admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.736309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a33c304-c43c-481f-9f10-0bcea348a65e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.736592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a33c304-c43c-481f-9f10-0bcea348a65e admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.737298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a33c304-c43c-481f-9f10-0bcea348a65e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:48.739207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a33c304-c43c-481f-9f10-0bcea348a65e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:48.766334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1225/2432] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:56:48 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:56:48.770887 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6535e850-014b-4e44-8ede-9f1fbfc8156d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:48.780904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6535e850-014b-4e44-8ede-9f1fbfc8156d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:48.836149 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f621b2-c20e-4d0d-b682-a1a9609c679f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:48.836796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f621b2-c20e-4d0d-b682-a1a9609c679f admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.837080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f621b2-c20e-4d0d-b682-a1a9609c679f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.837351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f621b2-c20e-4d0d-b682-a1a9609c679f admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.838007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f621b2-c20e-4d0d-b682-a1a9609c679f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:48.839918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f621b2-c20e-4d0d-b682-a1a9609c679f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:48.861057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1208/2433] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:56:48 2022] POST /identity/v3/users => generated 292 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:56:48.865170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a33c304-c43c-481f-9f10-0bcea348a65e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:48.875680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a33c304-c43c-481f-9f10-0bcea348a65e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:48.881172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912dd13f-32c0-4f9f-9a85-7c0902f63f2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:48.881779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912dd13f-32c0-4f9f-9a85-7c0902f63f2e admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.882064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912dd13f-32c0-4f9f-9a85-7c0902f63f2e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.882342 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912dd13f-32c0-4f9f-9a85-7c0902f63f2e admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.883032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912dd13f-32c0-4f9f-9a85-7c0902f63f2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:48.884686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912dd13f-32c0-4f9f-9a85-7c0902f63f2e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:48.895394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1226/2434] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:56:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:56:48.899209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f621b2-c20e-4d0d-b682-a1a9609c679f admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:48.909352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f621b2-c20e-4d0d-b682-a1a9609c679f admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:48.914414 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1886ba4f-7cba-45f3-bbdc-67e1d14ffbe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:48.915032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886ba4f-7cba-45f3-bbdc-67e1d14ffbe9 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.915319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886ba4f-7cba-45f3-bbdc-67e1d14ffbe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.915602 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886ba4f-7cba-45f3-bbdc-67e1d14ffbe9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.916335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1886ba4f-7cba-45f3-bbdc-67e1d14ffbe9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:48.917991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1886ba4f-7cba-45f3-bbdc-67e1d14ffbe9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:48.928809 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1209/2435] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:56:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:56:48.932589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912dd13f-32c0-4f9f-9a85-7c0902f63f2e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:48.945851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912dd13f-32c0-4f9f-9a85-7c0902f63f2e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:48.950641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e0131d-b50c-418a-a41e-764f61d54b20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:48.951348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e0131d-b50c-418a-a41e-764f61d54b20 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.951663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e0131d-b50c-418a-a41e-764f61d54b20 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.951948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e0131d-b50c-418a-a41e-764f61d54b20 admin admin] PATH_INFO: `/v3/projects/62319662d8e34fc8bbe01e0410cec30a/users/7136bb1746744ce89f74c1185c76d043/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.968149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e0131d-b50c-418a-a41e-764f61d54b20 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62319662d8e34fc8bbe01e0410cec30a, user_id=7136bb1746744ce89f74c1185c76d043, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:48.971208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e0131d-b50c-418a-a41e-764f61d54b20 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:48.983845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1227/2436] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:56:48 2022] PUT /identity/v3/projects/62319662d8e34fc8bbe01e0410cec30a/users/7136bb1746744ce89f74c1185c76d043/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:56:48.989269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc0edde-3e50-4787-a64e-db2b8d187ebd None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:48.989567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc0edde-3e50-4787-a64e-db2b8d187ebd None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:48.989855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc0edde-3e50-4787-a64e-db2b8d187ebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:48.997887 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:56:49.011510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-bcc0edde-3e50-4787-a64e-db2b8d187ebd None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:49.012734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-bcc0edde-3e50-4787-a64e-db2b8d187ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:49.071103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1210/2437] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:49.081376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e0131d-b50c-418a-a41e-764f61d54b20 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:49.083119 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-97e0131d-b50c-418a-a41e-764f61d54b20 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:49.139677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e0131d-b50c-418a-a41e-764f61d54b20 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:49.143488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a356ef1-cdd9-4480-b57d-d7274d3dc789 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:49.144090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a356ef1-cdd9-4480-b57d-d7274d3dc789 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:49.144382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a356ef1-cdd9-4480-b57d-d7274d3dc789 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:49.144693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a356ef1-cdd9-4480-b57d-d7274d3dc789 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:49.153601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a356ef1-cdd9-4480-b57d-d7274d3dc789 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:49.154981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a356ef1-cdd9-4480-b57d-d7274d3dc789 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:49.165303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1228/2438] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:56:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:56:50.021718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fe9abe-31e3-4511-a5e9-9077efb5d64d None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:50.022245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fe9abe-31e3-4511-a5e9-9077efb5d64d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:50.022345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fe9abe-31e3-4511-a5e9-9077efb5d64d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:50.050554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-05fe9abe-31e3-4511-a5e9-9077efb5d64d None None] MFA Rules not processed for user `7136bb1746744ce89f74c1185c76d043`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:50.051793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-05fe9abe-31e3-4511-a5e9-9077efb5d64d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:50.118119 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1211/2439] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:50 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:56:50.129852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a356ef1-cdd9-4480-b57d-d7274d3dc789 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:56:50.131313 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0a356ef1-cdd9-4480-b57d-d7274d3dc789 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:50.190486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a356ef1-cdd9-4480-b57d-d7274d3dc789 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:56:50.194148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fd39f9-951c-4f39-8e16-5dcea546acbd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:56:50.194769 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd39f9-951c-4f39-8e16-5dcea546acbd service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:50.195057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd39f9-951c-4f39-8e16-5dcea546acbd service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:50.195330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fd39f9-951c-4f39-8e16-5dcea546acbd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:50.204234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fd39f9-951c-4f39-8e16-5dcea546acbd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:56:50.205733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fd39f9-951c-4f39-8e16-5dcea546acbd service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:56:50.214932 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1229/2440] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:56:50 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:56:51.575141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0ac920-b93f-48e3-9a0e-0434327e811b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:56:51.575697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0ac920-b93f-48e3-9a0e-0434327e811b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:56:51.575848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0ac920-b93f-48e3-9a0e-0434327e811b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:56:51.598919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-fd0ac920-b93f-48e3-9a0e-0434327e811b None None] MFA Rules not processed for user `7136bb1746744ce89f74c1185c76d043`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:56:51.601224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fd0ac920-b93f-48e3-9a0e-0434327e811b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:56:51.609145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1212/2441] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:56:51 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:57:00.210719 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:57:01.591122 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=60 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:57:16.835647 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671efb01-2ed2-4bc4-a8a6-55549c424e03 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:57:16.836164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671efb01-2ed2-4bc4-a8a6-55549c424e03 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:57:16.836278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-671efb01-2ed2-4bc4-a8a6-55549c424e03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:57:16.866129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-671efb01-2ed2-4bc4-a8a6-55549c424e03 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:57:16.867360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-671efb01-2ed2-4bc4-a8a6-55549c424e03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:57:16.921238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1230/2442] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:57:16 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:57:16.969334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0ac920-b93f-48e3-9a0e-0434327e811b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:57:16.970771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fd0ac920-b93f-48e3-9a0e-0434327e811b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:57:17.043106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0ac920-b93f-48e3-9a0e-0434327e811b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:57:17.048146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5678bd-0809-4093-a172-98e58b8115bf service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:57:17.048814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5678bd-0809-4093-a172-98e58b8115bf service swift] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:57:17.049153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5678bd-0809-4093-a172-98e58b8115bf service swift] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:57:17.049462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5678bd-0809-4093-a172-98e58b8115bf service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:57:17.059229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5678bd-0809-4093-a172-98e58b8115bf service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:57:17.060863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5678bd-0809-4093-a172-98e58b8115bf service swift] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:57:17.070334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1213/2443] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:57:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:57:19.954643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671efb01-2ed2-4bc4-a8a6-55549c424e03 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:57:19.965130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-671efb01-2ed2-4bc4-a8a6-55549c424e03 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:57:19.969098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbf1254-a631-429e-b129-d6774fc513e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:57:19.969971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbf1254-a631-429e-b129-d6774fc513e0 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:57:19.970487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbf1254-a631-429e-b129-d6774fc513e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:57:19.970962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbf1254-a631-429e-b129-d6774fc513e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:57:19.973661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-dfbf1254-a631-429e-b129-d6774fc513e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:57:20.053325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbf1254-a631-429e-b129-d6774fc513e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:57:20.054902 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbf1254-a631-429e-b129-d6774fc513e0 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:57:20.066237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1231/2444] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:57:19 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:57:22.452858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5678bd-0809-4093-a172-98e58b8115bf service swift] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:57:22.454287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-cd5678bd-0809-4093-a172-98e58b8115bf service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:57:22.466309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5678bd-0809-4093-a172-98e58b8115bf service swift] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:57:22.469771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81f2f29-cd3c-49e1-89dd-4f3fede0e9a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:57:22.470378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f2f29-cd3c-49e1-89dd-4f3fede0e9a0 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:57:22.470657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f2f29-cd3c-49e1-89dd-4f3fede0e9a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:57:22.470928 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f2f29-cd3c-49e1-89dd-4f3fede0e9a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:57:22.473276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b81f2f29-cd3c-49e1-89dd-4f3fede0e9a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:57:22.487041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81f2f29-cd3c-49e1-89dd-4f3fede0e9a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:57:22.489283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81f2f29-cd3c-49e1-89dd-4f3fede0e9a0 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:57:22.500695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1214/2445] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:57:22 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:57:30.098997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=63 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:57:32.495839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=38 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:57:38.502134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f3efb0-848a-4e60-8e8f-548647221794 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:57:38.502637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f3efb0-848a-4e60-8e8f-548647221794 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:57:38.502732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f3efb0-848a-4e60-8e8f-548647221794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:57:38.522662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-66f3efb0-848a-4e60-8e8f-548647221794 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:57:38.523981 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-66f3efb0-848a-4e60-8e8f-548647221794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:57:38.531407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1232/2446] 10.222.0.40 () {58 vars in 1009 bytes} [Fri Jul 29 04:57:38 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:57:38.544714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81f2f29-cd3c-49e1-89dd-4f3fede0e9a0 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:57:38.556473 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81f2f29-cd3c-49e1-89dd-4f3fede0e9a0 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:57:38.561505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b860f86-8cd0-47c5-97a1-b0e89d2eef67 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:57:38.562225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b860f86-8cd0-47c5-97a1-b0e89d2eef67 service swift] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:57:38.562527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b860f86-8cd0-47c5-97a1-b0e89d2eef67 service swift] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:57:38.562807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b860f86-8cd0-47c5-97a1-b0e89d2eef67 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:57:38.572359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b860f86-8cd0-47c5-97a1-b0e89d2eef67 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:57:38.574085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b860f86-8cd0-47c5-97a1-b0e89d2eef67 service swift] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:57:38.583302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1215/2447] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:57:38 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:57:48.517165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:57:48.579624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=6 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:05.550553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f3efb0-848a-4e60-8e8f-548647221794 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:05.562674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f3efb0-848a-4e60-8e8f-548647221794 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:05.566340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c55dc0-a9ab-4ed3-9ed8-54ecc9f2542b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:05.567506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c55dc0-a9ab-4ed3-9ed8-54ecc9f2542b service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:05.567807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c55dc0-a9ab-4ed3-9ed8-54ecc9f2542b service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:05.568294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c55dc0-a9ab-4ed3-9ed8-54ecc9f2542b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:05.571336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-45c55dc0-a9ab-4ed3-9ed8-54ecc9f2542b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:05.582137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c55dc0-a9ab-4ed3-9ed8-54ecc9f2542b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:05.583588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c55dc0-a9ab-4ed3-9ed8-54ecc9f2542b service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:05.592863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1233/2448] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:05 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:07.834382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daff06d-93f6-4b80-bdf6-86fa9c8a3d69 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:07.835860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daff06d-93f6-4b80-bdf6-86fa9c8a3d69 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:07.836371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daff06d-93f6-4b80-bdf6-86fa9c8a3d69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:07.889797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-3daff06d-93f6-4b80-bdf6-86fa9c8a3d69 None None] MFA Rules not processed for user `3f53019d35f244e4959d8340ba6f7e45`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:07.891157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3daff06d-93f6-4b80-bdf6-86fa9c8a3d69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:07.910645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1216/2449] 10.222.0.40 () {58 vars in 967 bytes} [Fri Jul 29 04:58:07 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:07.925589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c55dc0-a9ab-4ed3-9ed8-54ecc9f2542b service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:07.951916 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c55dc0-a9ab-4ed3-9ed8-54ecc9f2542b service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:07.956951 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dba3ae-75b0-4bba-8f73-541dfa6c947f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bbfe92babca048d882cb59cb29b3ed97', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:07.957589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dba3ae-75b0-4bba-8f73-541dfa6c947f service swift] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:07.957875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dba3ae-75b0-4bba-8f73-541dfa6c947f service swift] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:07.958148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47dba3ae-75b0-4bba-8f73-541dfa6c947f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:07.966817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dba3ae-75b0-4bba-8f73-541dfa6c947f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:07.968378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47dba3ae-75b0-4bba-8f73-541dfa6c947f service swift] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:07.977458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1234/2450] 10.222.0.40 () {60 vars in 1384 bytes} [Fri Jul 29 04:58:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 29 04:58:17.907402 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=18 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:17.973193 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=16 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:24.398181 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86540ba-57ba-4bf0-9f8b-c6f0985fd249 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:24.398746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86540ba-57ba-4bf0-9f8b-c6f0985fd249 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:24.398746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86540ba-57ba-4bf0-9f8b-c6f0985fd249 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:24.418357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e86540ba-57ba-4bf0-9f8b-c6f0985fd249 None None] MFA Rules not processed for user `7136bb1746744ce89f74c1185c76d043`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:24.419571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e86540ba-57ba-4bf0-9f8b-c6f0985fd249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:24.426448 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1217/2451] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:24.984428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dba3ae-75b0-4bba-8f73-541dfa6c947f service swift] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:24.985999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-47dba3ae-75b0-4bba-8f73-541dfa6c947f service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:25.074940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47dba3ae-75b0-4bba-8f73-541dfa6c947f service swift] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:25.080144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1636f0f-c612-48cf-aa3e-1afc611ff8f7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:25.080812 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1636f0f-c612-48cf-aa3e-1afc611ff8f7 service placement] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:25.081105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1636f0f-c612-48cf-aa3e-1afc611ff8f7 service placement] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:25.081388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1636f0f-c612-48cf-aa3e-1afc611ff8f7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:25.083186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c1636f0f-c612-48cf-aa3e-1afc611ff8f7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:25.094835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1636f0f-c612-48cf-aa3e-1afc611ff8f7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:25.096421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1636f0f-c612-48cf-aa3e-1afc611ff8f7 service placement] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:25.107236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1235/2452] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:24 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:26.175723 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86540ba-57ba-4bf0-9f8b-c6f0985fd249 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:26.187420 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86540ba-57ba-4bf0-9f8b-c6f0985fd249 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:26.191612 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bfc0d2-c556-424c-ba82-dde1e5236685 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:26.192262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfc0d2-c556-424c-ba82-dde1e5236685 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:26.192572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfc0d2-c556-424c-ba82-dde1e5236685 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:26.192871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfc0d2-c556-424c-ba82-dde1e5236685 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:26.195292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c9bfc0d2-c556-424c-ba82-dde1e5236685 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:26.207351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9bfc0d2-c556-424c-ba82-dde1e5236685 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:26.208863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9bfc0d2-c556-424c-ba82-dde1e5236685 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:26.219733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1218/2453] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:31.263363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1636f0f-c612-48cf-aa3e-1afc611ff8f7 service placement] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:31.265384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c1636f0f-c612-48cf-aa3e-1afc611ff8f7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:31.280677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1636f0f-c612-48cf-aa3e-1afc611ff8f7 service placement] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:31.285169 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:31.285817 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:31.286280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:31.286705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] PATH_INFO: `/v3/users/8b7751b93f88424abebe6859f46f8351` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:31.302761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b7751b93f88424abebe6859f46f8351)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:31.305703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:31.355903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b7751b93f88424abebe6859f46f8351'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:31.356569 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b7751b93f88424abebe6859f46f8351'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:31.357208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b7751b93f88424abebe6859f46f8351'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:31.369200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b7751b93f88424abebe6859f46f8351'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:31.376246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6497dd26-2647-4eb7-91f1-e4ba4b982a7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b7751b93f88424abebe6859f46f8351'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:31.389308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1236/2454] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:58:31 2022] DELETE /identity/v3/users/8b7751b93f88424abebe6859f46f8351 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:31.566325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bfc0d2-c556-424c-ba82-dde1e5236685 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:31.577738 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bfc0d2-c556-424c-ba82-dde1e5236685 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:31.635651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:31.636357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:31.636670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:31.636950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] PATH_INFO: `/v3/projects/946bf1e173f9476d86cf6133055fba0a` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:31.647113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=946bf1e173f9476d86cf6133055fba0a)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:31.649103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:31.695490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '946bf1e173f9476d86cf6133055fba0a', 'request_id': 'req-f381210e-f7ff-462b-a985-7e0267f3d1ca'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:31.702714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '946bf1e173f9476d86cf6133055fba0a', 'request_id': 'req-f381210e-f7ff-462b-a985-7e0267f3d1ca'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:31.708226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '946bf1e173f9476d86cf6133055fba0a', 'request_id': 'req-f381210e-f7ff-462b-a985-7e0267f3d1ca'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:31.719606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] The token cache is being invalidate because project 946bf1e173f9476d86cf6133055fba0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:58:31.719896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:31.720553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f381210e-f7ff-462b-a985-7e0267f3d1ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:31.722080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1219/2455] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:58:31 2022] DELETE /identity/v3/projects/946bf1e173f9476d86cf6133055fba0a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:31.727604 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a570e7-fba3-4957-86b9-4ac98b51e915 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:31.727903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a570e7-fba3-4957-86b9-4ac98b51e915 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:31.728194 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a570e7-fba3-4957-86b9-4ac98b51e915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:31.748141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-e9a570e7-fba3-4957-86b9-4ac98b51e915 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:31.749401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e9a570e7-fba3-4957-86b9-4ac98b51e915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:31.794765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:31.811172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1237/2456] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:31.820742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d7ca5e-d83c-4418-a1fc-1f884a4cd47f None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:31.820944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d7ca5e-d83c-4418-a1fc-1f884a4cd47f None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:31.821235 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d7ca5e-d83c-4418-a1fc-1f884a4cd47f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:31.843340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-25d7ca5e-d83c-4418-a1fc-1f884a4cd47f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:31.844588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-25d7ca5e-d83c-4418-a1fc-1f884a4cd47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:31.857478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1220/2457] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:31.861692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a570e7-fba3-4957-86b9-4ac98b51e915 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:31.871567 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a570e7-fba3-4957-86b9-4ac98b51e915 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:31.878906 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5346fa-d392-48a5-9065-a323bae271f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:31.879614 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5346fa-d392-48a5-9065-a323bae271f9 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:31.879844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5346fa-d392-48a5-9065-a323bae271f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:31.880150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5346fa-d392-48a5-9065-a323bae271f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:31.880955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5346fa-d392-48a5-9065-a323bae271f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:58:31.881299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5346fa-d392-48a5-9065-a323bae271f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:31.889325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5346fa-d392-48a5-9065-a323bae271f9 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:31.904642 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1238/2458] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:58:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:58:31.907512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d7ca5e-d83c-4418-a1fc-1f884a4cd47f None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:31.919636 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d7ca5e-d83c-4418-a1fc-1f884a4cd47f None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:31.924512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be05366-579c-4b59-bee9-1a040915a1fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:31.925251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be05366-579c-4b59-bee9-1a040915a1fb admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:31.925609 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be05366-579c-4b59-bee9-1a040915a1fb admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:31.925961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be05366-579c-4b59-bee9-1a040915a1fb admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:31.926839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be05366-579c-4b59-bee9-1a040915a1fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:31.929665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be05366-579c-4b59-bee9-1a040915a1fb admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:31.973388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1221/2459] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:58:31 2022] POST /identity/v3/projects => generated 381 bytes in 66 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:58:31.977856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5346fa-d392-48a5-9065-a323bae271f9 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:31.988024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5346fa-d392-48a5-9065-a323bae271f9 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.046778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3fa960-c80d-477e-83f5-8ffac0674b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.047317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3fa960-c80d-477e-83f5-8ffac0674b82 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.047607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3fa960-c80d-477e-83f5-8ffac0674b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.047887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3fa960-c80d-477e-83f5-8ffac0674b82 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.049233 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3fa960-c80d-477e-83f5-8ffac0674b82 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.051192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3fa960-c80d-477e-83f5-8ffac0674b82 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.076923 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1239/2460] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:58:31 2022] POST /identity/v3/users => generated 291 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:58:32.081127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be05366-579c-4b59-bee9-1a040915a1fb admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:32.091988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be05366-579c-4b59-bee9-1a040915a1fb admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.096907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3d81bd-94bc-48b3-a092-b3f21a3e46a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.097519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3d81bd-94bc-48b3-a092-b3f21a3e46a6 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.097803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3d81bd-94bc-48b3-a092-b3f21a3e46a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.098078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3d81bd-94bc-48b3-a092-b3f21a3e46a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.098776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3d81bd-94bc-48b3-a092-b3f21a3e46a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.100551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3d81bd-94bc-48b3-a092-b3f21a3e46a6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.110849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1222/2461] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:32.114816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3fa960-c80d-477e-83f5-8ffac0674b82 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:32.125438 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3fa960-c80d-477e-83f5-8ffac0674b82 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.130061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bf34bf-04e4-4fc9-a6c9-f46d79c658ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.130671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bf34bf-04e4-4fc9-a6c9-f46d79c658ad admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.130962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bf34bf-04e4-4fc9-a6c9-f46d79c658ad admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.131243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bf34bf-04e4-4fc9-a6c9-f46d79c658ad admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.132595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bf34bf-04e4-4fc9-a6c9-f46d79c658ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.134253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24bf34bf-04e4-4fc9-a6c9-f46d79c658ad admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.145681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1240/2462] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:58:32.149770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3d81bd-94bc-48b3-a092-b3f21a3e46a6 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:32.159680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3d81bd-94bc-48b3-a092-b3f21a3e46a6 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.164222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbac6ad-6866-40d1-86a9-351bf1cd8550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.164906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbac6ad-6866-40d1-86a9-351bf1cd8550 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.165193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbac6ad-6866-40d1-86a9-351bf1cd8550 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.165470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbac6ad-6866-40d1-86a9-351bf1cd8550 admin admin] PATH_INFO: `/v3/projects/4960e45e0f4f4bab92a3dbf7492ba652/users/b23e0f6304e5445d8c16fcb6b91eddf3/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.176659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:32.181622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbac6ad-6866-40d1-86a9-351bf1cd8550 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4960e45e0f4f4bab92a3dbf7492ba652, user_id=b23e0f6304e5445d8c16fcb6b91eddf3, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.184684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbac6ad-6866-40d1-86a9-351bf1cd8550 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.198780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1223/2463] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:58:32 2022] PUT /identity/v3/projects/4960e45e0f4f4bab92a3dbf7492ba652/users/b23e0f6304e5445d8c16fcb6b91eddf3/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:32.208881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a109206-3477-46b5-b247-0a2b3af8b109 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.209175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a109206-3477-46b5-b247-0a2b3af8b109 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.209454 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a109206-3477-46b5-b247-0a2b3af8b109 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.236953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2a109206-3477-46b5-b247-0a2b3af8b109 None None] MFA Rules not processed for user `b23e0f6304e5445d8c16fcb6b91eddf3`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:32.238183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2a109206-3477-46b5-b247-0a2b3af8b109 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:32.302662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1241/2464] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:32 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:32.307708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbac6ad-6866-40d1-86a9-351bf1cd8550 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:32.371470 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbac6ad-6866-40d1-86a9-351bf1cd8550 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.376089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943669dc-0cfe-4e31-a5c6-fbeea4f9c20e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.376722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943669dc-0cfe-4e31-a5c6-fbeea4f9c20e admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.377011 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943669dc-0cfe-4e31-a5c6-fbeea4f9c20e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.377291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943669dc-0cfe-4e31-a5c6-fbeea4f9c20e admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.377945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943669dc-0cfe-4e31-a5c6-fbeea4f9c20e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.379839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943669dc-0cfe-4e31-a5c6-fbeea4f9c20e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.408257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1224/2465] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:58:32 2022] POST /identity/v3/projects => generated 383 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:58:32.412525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a109206-3477-46b5-b247-0a2b3af8b109 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:32.423498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a109206-3477-46b5-b247-0a2b3af8b109 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.466942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:32.479905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc6b0b-96c6-464e-b26f-6ebae9353827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.480530 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc6b0b-96c6-464e-b26f-6ebae9353827 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.480848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc6b0b-96c6-464e-b26f-6ebae9353827 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.481130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cc6b0b-96c6-464e-b26f-6ebae9353827 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.481807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cc6b0b-96c6-464e-b26f-6ebae9353827 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.483733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cc6b0b-96c6-464e-b26f-6ebae9353827 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.505105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1242/2466] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:58:32 2022] POST /identity/v3/users => generated 298 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:58:32.509603 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943669dc-0cfe-4e31-a5c6-fbeea4f9c20e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:32.519701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943669dc-0cfe-4e31-a5c6-fbeea4f9c20e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.525053 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83c0fbb-44e8-427f-b639-9473f3215a4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.525661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83c0fbb-44e8-427f-b639-9473f3215a4e admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.525942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83c0fbb-44e8-427f-b639-9473f3215a4e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.526217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83c0fbb-44e8-427f-b639-9473f3215a4e admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.526906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83c0fbb-44e8-427f-b639-9473f3215a4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.528493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83c0fbb-44e8-427f-b639-9473f3215a4e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.538812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1225/2467] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:32.543061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc6b0b-96c6-464e-b26f-6ebae9353827 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:32.553055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cc6b0b-96c6-464e-b26f-6ebae9353827 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.557828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7f374c-401c-4771-9ca8-00077e293138 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.558517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7f374c-401c-4771-9ca8-00077e293138 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.558801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7f374c-401c-4771-9ca8-00077e293138 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.559079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7f374c-401c-4771-9ca8-00077e293138 admin admin] PATH_INFO: `/v3/projects/a09bd872122f4e59b1973959682311eb/users/a0dfcdd4f17641269a794f35a91f3beb/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.574112 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7f374c-401c-4771-9ca8-00077e293138 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a09bd872122f4e59b1973959682311eb, user_id=a0dfcdd4f17641269a794f35a91f3beb, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.576839 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7f374c-401c-4771-9ca8-00077e293138 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.590527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1243/2468] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:58:32 2022] PUT /identity/v3/projects/a09bd872122f4e59b1973959682311eb/users/a0dfcdd4f17641269a794f35a91f3beb/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:32.595196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83c0fbb-44e8-427f-b639-9473f3215a4e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:32.605754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83c0fbb-44e8-427f-b639-9473f3215a4e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.666920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa41457e-31ef-418c-98b0-cfd45c26ca9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.667593 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa41457e-31ef-418c-98b0-cfd45c26ca9d admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.667877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa41457e-31ef-418c-98b0-cfd45c26ca9d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.668154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa41457e-31ef-418c-98b0-cfd45c26ca9d admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.668887 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa41457e-31ef-418c-98b0-cfd45c26ca9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.670495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa41457e-31ef-418c-98b0-cfd45c26ca9d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.681460 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1226/2469] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:32.685409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7f374c-401c-4771-9ca8-00077e293138 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:32.696568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7f374c-401c-4771-9ca8-00077e293138 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.701207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383d93f2-6a4e-4f85-8188-ec8c39b8bf75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.702017 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d93f2-6a4e-4f85-8188-ec8c39b8bf75 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.702304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d93f2-6a4e-4f85-8188-ec8c39b8bf75 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.702588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383d93f2-6a4e-4f85-8188-ec8c39b8bf75 admin admin] PATH_INFO: `/v3/projects/a09bd872122f4e59b1973959682311eb/users/a0dfcdd4f17641269a794f35a91f3beb/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.704672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383d93f2-6a4e-4f85-8188-ec8c39b8bf75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a09bd872122f4e59b1973959682311eb, user_id=a0dfcdd4f17641269a794f35a91f3beb, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.707319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383d93f2-6a4e-4f85-8188-ec8c39b8bf75 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.718598 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1244/2470] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:58:32 2022] PUT /identity/v3/projects/a09bd872122f4e59b1973959682311eb/users/a0dfcdd4f17641269a794f35a91f3beb/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:32.736069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0dde39-2ed8-4400-86ec-0ce2fa759499 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.736492 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0dde39-2ed8-4400-86ec-0ce2fa759499 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.736898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0dde39-2ed8-4400-86ec-0ce2fa759499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.766933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-aa0dde39-2ed8-4400-86ec-0ce2fa759499 None None] MFA Rules not processed for user `a0dfcdd4f17641269a794f35a91f3beb`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:32.768207 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-aa0dde39-2ed8-4400-86ec-0ce2fa759499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:32.817293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:32.841910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1227/2471] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:32 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:32.852910 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383d93f2-6a4e-4f85-8188-ec8c39b8bf75 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:32.854703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-383d93f2-6a4e-4f85-8188-ec8c39b8bf75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:32.913229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383d93f2-6a4e-4f85-8188-ec8c39b8bf75 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:32.917363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a04521d-2939-4d13-8bf2-9cc09e91208f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:32.917980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04521d-2939-4d13-8bf2-9cc09e91208f service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:32.918266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04521d-2939-4d13-8bf2-9cc09e91208f service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:32.918548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04521d-2939-4d13-8bf2-9cc09e91208f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:32.928433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a04521d-2939-4d13-8bf2-9cc09e91208f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:32.930456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a04521d-2939-4d13-8bf2-9cc09e91208f service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:32.940703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1245/2472] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:32 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:33.045013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0dde39-2ed8-4400-86ec-0ce2fa759499 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:33.056130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0dde39-2ed8-4400-86ec-0ce2fa759499 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:33.059780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9a22bb-c799-453b-9396-13717cf978e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:33.060391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a22bb-c799-453b-9396-13717cf978e7 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:33.060713 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a22bb-c799-453b-9396-13717cf978e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:33.060989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9a22bb-c799-453b-9396-13717cf978e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:33.121975 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9a22bb-c799-453b-9396-13717cf978e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:33.124218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9a22bb-c799-453b-9396-13717cf978e7 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:33.134919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1228/2473] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:33 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:35.722246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a04521d-2939-4d13-8bf2-9cc09e91208f service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:35.782730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a04521d-2939-4d13-8bf2-9cc09e91208f service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:35.787246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:35.787872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:35.788311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:35.788849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] PATH_INFO: `/v3/users/b23e0f6304e5445d8c16fcb6b91eddf3` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:35.789701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b23e0f6304e5445d8c16fcb6b91eddf3)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:35.791857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:35.841559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:35.844362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b23e0f6304e5445d8c16fcb6b91eddf3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:35.845525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b23e0f6304e5445d8c16fcb6b91eddf3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:35.846275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b23e0f6304e5445d8c16fcb6b91eddf3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:35.860526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b23e0f6304e5445d8c16fcb6b91eddf3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:35.868023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b23e0f6304e5445d8c16fcb6b91eddf3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:35.880834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1246/2474] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:58:35 2022] DELETE /identity/v3/users/b23e0f6304e5445d8c16fcb6b91eddf3 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:35.885990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ca839-c2da-4c8d-ac0f-b7e22bf1763c None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:35.886444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ca839-c2da-4c8d-ac0f-b7e22bf1763c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:35.886554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ca839-c2da-4c8d-ac0f-b7e22bf1763c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:35.905387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-7a7ca839-c2da-4c8d-ac0f-b7e22bf1763c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:35.906615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7a7ca839-c2da-4c8d-ac0f-b7e22bf1763c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:35.964578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1229/2475] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:35.974091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:35.975526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:36.032151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec60aa1-50e9-4c7e-ab5c-20dfd5204af6 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:36.036016 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea02ca5-4f30-4f16-9875-ce789ff46017 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:36.036685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea02ca5-4f30-4f16-9875-ce789ff46017 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:36.036978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea02ca5-4f30-4f16-9875-ce789ff46017 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:36.037263 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea02ca5-4f30-4f16-9875-ce789ff46017 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:36.047500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea02ca5-4f30-4f16-9875-ce789ff46017 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:36.048969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea02ca5-4f30-4f16-9875-ce789ff46017 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:36.059402 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1247/2476] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:36.196683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ca839-c2da-4c8d-ac0f-b7e22bf1763c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:36.198112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7a7ca839-c2da-4c8d-ac0f-b7e22bf1763c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:36.307988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ca839-c2da-4c8d-ac0f-b7e22bf1763c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:36.313816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2072b662-347e-4f54-ac8c-4dd925c8ea94 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:36.314446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2072b662-347e-4f54-ac8c-4dd925c8ea94 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:36.314734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2072b662-347e-4f54-ac8c-4dd925c8ea94 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:36.315014 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2072b662-347e-4f54-ac8c-4dd925c8ea94 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:36.316815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2072b662-347e-4f54-ac8c-4dd925c8ea94 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:36.330972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2072b662-347e-4f54-ac8c-4dd925c8ea94 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:36.332622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2072b662-347e-4f54-ac8c-4dd925c8ea94 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:36.344485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1230/2477] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:36 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:36.356050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea02ca5-4f30-4f16-9875-ce789ff46017 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:36.357620 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-9ea02ca5-4f30-4f16-9875-ce789ff46017 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:36.372844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea02ca5-4f30-4f16-9875-ce789ff46017 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:36.377735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:36.379184 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:36.379593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:36.379814 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] PATH_INFO: `/v3/projects/4960e45e0f4f4bab92a3dbf7492ba652` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:36.381134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4960e45e0f4f4bab92a3dbf7492ba652)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:36.383790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:36.433355 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4960e45e0f4f4bab92a3dbf7492ba652', 'request_id': 'req-913d4bb0-c880-463e-8dce-08e22cbed5e6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:36.439712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4960e45e0f4f4bab92a3dbf7492ba652', 'request_id': 'req-913d4bb0-c880-463e-8dce-08e22cbed5e6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:36.448186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4960e45e0f4f4bab92a3dbf7492ba652', 'request_id': 'req-913d4bb0-c880-463e-8dce-08e22cbed5e6'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:36.458945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] The token cache is being invalidate because project 4960e45e0f4f4bab92a3dbf7492ba652 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:58:36.459245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:36.460026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:36.461865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1248/2478] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:58:36 2022] DELETE /identity/v3/projects/4960e45e0f4f4bab92a3dbf7492ba652 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:36.465807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2072b662-347e-4f54-ac8c-4dd925c8ea94 service nova] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:36.476774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2072b662-347e-4f54-ac8c-4dd925c8ea94 service nova] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:36.482060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=100 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:36.538241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:36.538898 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:36.539180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:36.539462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] PATH_INFO: `/v3/users/a0dfcdd4f17641269a794f35a91f3beb` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:36.541515 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a0dfcdd4f17641269a794f35a91f3beb)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:36.543474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:36.591046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a0dfcdd4f17641269a794f35a91f3beb'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:36.598471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a0dfcdd4f17641269a794f35a91f3beb'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:36.599102 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a0dfcdd4f17641269a794f35a91f3beb'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:36.599767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a0dfcdd4f17641269a794f35a91f3beb'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:36.607543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-87773f6b-d579-4b05-8e8d-7eaa6f63a778 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a0dfcdd4f17641269a794f35a91f3beb'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:36.637154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1231/2479] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:58:36 2022] DELETE /identity/v3/users/a0dfcdd4f17641269a794f35a91f3beb => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:58:36.910025 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:36.912093 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:36.932058 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913d4bb0-c880-463e-8dce-08e22cbed5e6 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:36.977116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:DELETE=11,keystone:SELECT=87,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:36.992275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:36.993091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:36.993380 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:36.993668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] PATH_INFO: `/v3/projects/a09bd872122f4e59b1973959682311eb` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:36.994855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a09bd872122f4e59b1973959682311eb)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:36.996852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.045495 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a09bd872122f4e59b1973959682311eb', 'request_id': 'req-a7f0c7b2-498f-415d-a462-7509a7bb0523'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:37.051499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a09bd872122f4e59b1973959682311eb', 'request_id': 'req-a7f0c7b2-498f-415d-a462-7509a7bb0523'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:37.063224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a09bd872122f4e59b1973959682311eb', 'request_id': 'req-a7f0c7b2-498f-415d-a462-7509a7bb0523'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:37.074516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] The token cache is being invalidate because project a09bd872122f4e59b1973959682311eb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:58:37.074809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:37.075465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a7f0c7b2-498f-415d-a462-7509a7bb0523 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:37.077047 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1249/2480] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:58:36 2022] DELETE /identity/v3/projects/a09bd872122f4e59b1973959682311eb => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:37.081862 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aab3b6c-3797-4c75-b80c-4283fcbd588a None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.082361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aab3b6c-3797-4c75-b80c-4283fcbd588a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.082466 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aab3b6c-3797-4c75-b80c-4283fcbd588a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.103228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-7aab3b6c-3797-4c75-b80c-4283fcbd588a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:37.104441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7aab3b6c-3797-4c75-b80c-4283fcbd588a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:37.166446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1232/2481] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:37.174641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597bf82f-198a-4a8d-955c-c683287ea43e None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.175011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597bf82f-198a-4a8d-955c-c683287ea43e None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.175408 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597bf82f-198a-4a8d-955c-c683287ea43e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.196213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-597bf82f-198a-4a8d-955c-c683287ea43e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:37.197505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-597bf82f-198a-4a8d-955c-c683287ea43e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:37.205836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1250/2482] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:37.210895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aab3b6c-3797-4c75-b80c-4283fcbd588a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.220660 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aab3b6c-3797-4c75-b80c-4283fcbd588a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.225193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d088e1-e023-4669-8187-4d6ed3152382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.226425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d088e1-e023-4669-8187-4d6ed3152382 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.226807 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d088e1-e023-4669-8187-4d6ed3152382 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.227096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d088e1-e023-4669-8187-4d6ed3152382 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.227878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d088e1-e023-4669-8187-4d6ed3152382 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:58:37.228135 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d088e1-e023-4669-8187-4d6ed3152382 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.229785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d088e1-e023-4669-8187-4d6ed3152382 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.241444 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1233/2483] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:58:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:58:37.245425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597bf82f-198a-4a8d-955c-c683287ea43e None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.254644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597bf82f-198a-4a8d-955c-c683287ea43e None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.259596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f01a17-32d2-4c47-bee0-eea3e11ad36f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.260202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f01a17-32d2-4c47-bee0-eea3e11ad36f admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.260490 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f01a17-32d2-4c47-bee0-eea3e11ad36f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.260802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f01a17-32d2-4c47-bee0-eea3e11ad36f admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.261507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f01a17-32d2-4c47-bee0-eea3e11ad36f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.263703 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f01a17-32d2-4c47-bee0-eea3e11ad36f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.292080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1251/2484] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:58:37 2022] POST /identity/v3/projects => generated 383 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:58:37.296139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d088e1-e023-4669-8187-4d6ed3152382 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.307088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d088e1-e023-4669-8187-4d6ed3152382 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.318454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:37.364626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c7ec4a-6b48-4b50-ae84-c90dcdbea449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.365257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c7ec4a-6b48-4b50-ae84-c90dcdbea449 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.365540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c7ec4a-6b48-4b50-ae84-c90dcdbea449 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.365815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c7ec4a-6b48-4b50-ae84-c90dcdbea449 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.366655 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c7ec4a-6b48-4b50-ae84-c90dcdbea449 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.368490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c7ec4a-6b48-4b50-ae84-c90dcdbea449 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.389180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1234/2485] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:58:37 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:58:37.393680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f01a17-32d2-4c47-bee0-eea3e11ad36f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.405445 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f01a17-32d2-4c47-bee0-eea3e11ad36f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.410324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f9b86e-0a39-4036-ace0-fd6230f3d54d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.410941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f9b86e-0a39-4036-ace0-fd6230f3d54d admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.411230 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f9b86e-0a39-4036-ace0-fd6230f3d54d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.411514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f9b86e-0a39-4036-ace0-fd6230f3d54d admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.412212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f9b86e-0a39-4036-ace0-fd6230f3d54d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.414647 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f9b86e-0a39-4036-ace0-fd6230f3d54d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.424961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1252/2486] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:58:37.428962 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c7ec4a-6b48-4b50-ae84-c90dcdbea449 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.438914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c7ec4a-6b48-4b50-ae84-c90dcdbea449 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.443503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ab494-9ab9-49da-a809-43c62177b087 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.444125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ab494-9ab9-49da-a809-43c62177b087 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.444449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ab494-9ab9-49da-a809-43c62177b087 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.444805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ab494-9ab9-49da-a809-43c62177b087 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.445472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ab494-9ab9-49da-a809-43c62177b087 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.447073 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ab494-9ab9-49da-a809-43c62177b087 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.457633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1235/2487] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:37 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:37.462209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f9b86e-0a39-4036-ace0-fd6230f3d54d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.472244 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f9b86e-0a39-4036-ace0-fd6230f3d54d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.477537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f085c01a-9e63-47c1-a062-0f1158fa169f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.478249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085c01a-9e63-47c1-a062-0f1158fa169f admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.478486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085c01a-9e63-47c1-a062-0f1158fa169f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.478768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085c01a-9e63-47c1-a062-0f1158fa169f admin admin] PATH_INFO: `/v3/projects/aedc46077c3d41ee9f7f9e80a8af9f27/users/cad24748e280484fa30f081696d308b8/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.493359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f085c01a-9e63-47c1-a062-0f1158fa169f admin admin] RBAC: Authorizing `identity:create_grant(project_id=aedc46077c3d41ee9f7f9e80a8af9f27, user_id=cad24748e280484fa30f081696d308b8, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.496315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f085c01a-9e63-47c1-a062-0f1158fa169f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.509649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1253/2488] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:58:37 2022] PUT /identity/v3/projects/aedc46077c3d41ee9f7f9e80a8af9f27/users/cad24748e280484fa30f081696d308b8/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:37.519986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a2c6ac-2a96-4dec-82d6-4b3676ae2ba5 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.520283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a2c6ac-2a96-4dec-82d6-4b3676ae2ba5 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.520570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a2c6ac-2a96-4dec-82d6-4b3676ae2ba5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.549396 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-02a2c6ac-2a96-4dec-82d6-4b3676ae2ba5 None None] MFA Rules not processed for user `cad24748e280484fa30f081696d308b8`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:37.550614 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-02a2c6ac-2a96-4dec-82d6-4b3676ae2ba5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:37.614169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1236/2489] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:37 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:37.618743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f085c01a-9e63-47c1-a062-0f1158fa169f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.681321 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f085c01a-9e63-47c1-a062-0f1158fa169f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.686223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d451718-e784-48cf-b458-af36a8eb5e3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.686875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d451718-e784-48cf-b458-af36a8eb5e3d admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.687167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d451718-e784-48cf-b458-af36a8eb5e3d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.687448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d451718-e784-48cf-b458-af36a8eb5e3d admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.688131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d451718-e784-48cf-b458-af36a8eb5e3d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.690110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d451718-e784-48cf-b458-af36a8eb5e3d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.711403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:37.719241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1254/2490] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:58:37 2022] POST /identity/v3/projects => generated 381 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:58:37.723297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a2c6ac-2a96-4dec-82d6-4b3676ae2ba5 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.734143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a2c6ac-2a96-4dec-82d6-4b3676ae2ba5 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.790847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c2054a-ea61-429f-9788-21c69b05b2e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.791733 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c2054a-ea61-429f-9788-21c69b05b2e3 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.792016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c2054a-ea61-429f-9788-21c69b05b2e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.792299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c2054a-ea61-429f-9788-21c69b05b2e3 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.792991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c2054a-ea61-429f-9788-21c69b05b2e3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.794906 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c2054a-ea61-429f-9788-21c69b05b2e3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.819179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1237/2491] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:58:37 2022] POST /identity/v3/users => generated 297 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:58:37.821238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d451718-e784-48cf-b458-af36a8eb5e3d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.832690 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d451718-e784-48cf-b458-af36a8eb5e3d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.837988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a53f3-b040-404a-8bec-38467560cfc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.838608 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a53f3-b040-404a-8bec-38467560cfc4 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.838894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a53f3-b040-404a-8bec-38467560cfc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.839304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a53f3-b040-404a-8bec-38467560cfc4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.840011 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a53f3-b040-404a-8bec-38467560cfc4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.841654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a53f3-b040-404a-8bec-38467560cfc4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.853023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1255/2492] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:37 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:37.857048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c2054a-ea61-429f-9788-21c69b05b2e3 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.869859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c2054a-ea61-429f-9788-21c69b05b2e3 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.874577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30850ab-2cdb-4004-82f7-565b7aeaa320 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.875257 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30850ab-2cdb-4004-82f7-565b7aeaa320 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.875978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30850ab-2cdb-4004-82f7-565b7aeaa320 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.875978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30850ab-2cdb-4004-82f7-565b7aeaa320 admin admin] PATH_INFO: `/v3/projects/50af6523a3b04b8a962964f59e890d57/users/b09ff3085e7a4008af47ba98b4179b7b/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.888324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:37.892288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30850ab-2cdb-4004-82f7-565b7aeaa320 admin admin] RBAC: Authorizing `identity:create_grant(project_id=50af6523a3b04b8a962964f59e890d57, user_id=b09ff3085e7a4008af47ba98b4179b7b, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.895620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30850ab-2cdb-4004-82f7-565b7aeaa320 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.909754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1238/2493] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:58:37 2022] PUT /identity/v3/projects/50af6523a3b04b8a962964f59e890d57/users/b09ff3085e7a4008af47ba98b4179b7b/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:37.913747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a53f3-b040-404a-8bec-38467560cfc4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:37.924094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a53f3-b040-404a-8bec-38467560cfc4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:37.981871 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f949d4e-53bc-4410-9992-9fbf5f3b09f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:37.982488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f949d4e-53bc-4410-9992-9fbf5f3b09f3 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:37.982774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f949d4e-53bc-4410-9992-9fbf5f3b09f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:37.983053 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f949d4e-53bc-4410-9992-9fbf5f3b09f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:37.983743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f949d4e-53bc-4410-9992-9fbf5f3b09f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:37.985427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f949d4e-53bc-4410-9992-9fbf5f3b09f3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:37.995502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1256/2494] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:37 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:58:37.999759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30850ab-2cdb-4004-82f7-565b7aeaa320 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:38.010437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30850ab-2cdb-4004-82f7-565b7aeaa320 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:38.014942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071861c5-e892-425b-896a-250998887d1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:38.015596 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071861c5-e892-425b-896a-250998887d1d admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:38.015879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071861c5-e892-425b-896a-250998887d1d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:38.016158 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071861c5-e892-425b-896a-250998887d1d admin admin] PATH_INFO: `/v3/projects/50af6523a3b04b8a962964f59e890d57/users/b09ff3085e7a4008af47ba98b4179b7b/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:38.018117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071861c5-e892-425b-896a-250998887d1d admin admin] RBAC: Authorizing `identity:create_grant(project_id=50af6523a3b04b8a962964f59e890d57, user_id=b09ff3085e7a4008af47ba98b4179b7b, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:38.020770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071861c5-e892-425b-896a-250998887d1d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:38.030917 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1239/2495] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:58:37 2022] PUT /identity/v3/projects/50af6523a3b04b8a962964f59e890d57/users/b09ff3085e7a4008af47ba98b4179b7b/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:38.042310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e530b5-90b5-4d64-a9ac-3ca0ca71f722 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:38.042675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e530b5-90b5-4d64-a9ac-3ca0ca71f722 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:38.042992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e530b5-90b5-4d64-a9ac-3ca0ca71f722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:38.071150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c2e530b5-90b5-4d64-a9ac-3ca0ca71f722 None None] MFA Rules not processed for user `b09ff3085e7a4008af47ba98b4179b7b`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:38.072359 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c2e530b5-90b5-4d64-a9ac-3ca0ca71f722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:38.139644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1257/2496] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:38 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:38.151338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071861c5-e892-425b-896a-250998887d1d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:38.153233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-071861c5-e892-425b-896a-250998887d1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:38.215474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071861c5-e892-425b-896a-250998887d1d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:38.219238 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1ef7ca-6ac8-4833-8f7d-58d31f1775d3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:38.219865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ef7ca-6ac8-4833-8f7d-58d31f1775d3 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:38.220151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ef7ca-6ac8-4833-8f7d-58d31f1775d3 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:38.220431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1ef7ca-6ac8-4833-8f7d-58d31f1775d3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:38.229480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1ef7ca-6ac8-4833-8f7d-58d31f1775d3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:38.230921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1ef7ca-6ac8-4833-8f7d-58d31f1775d3 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:38.240845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1240/2497] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:38 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:38.315148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e530b5-90b5-4d64-a9ac-3ca0ca71f722 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:38.325297 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e530b5-90b5-4d64-a9ac-3ca0ca71f722 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:38.328972 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8da856-beef-4959-8468-98a58a25ed35 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:38.329590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8da856-beef-4959-8468-98a58a25ed35 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:38.329877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8da856-beef-4959-8468-98a58a25ed35 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:38.330155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8da856-beef-4959-8468-98a58a25ed35 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:38.386515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8da856-beef-4959-8468-98a58a25ed35 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:38.387962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8da856-beef-4959-8468-98a58a25ed35 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:38.398190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1258/2498] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:38 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:38.672989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1ef7ca-6ac8-4833-8f7d-58d31f1775d3 service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:38.750141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1ef7ca-6ac8-4833-8f7d-58d31f1775d3 service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:38.754748 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:38.755398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:38.755684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:38.756479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] PATH_INFO: `/v3/users/cad24748e280484fa30f081696d308b8` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:38.757245 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=cad24748e280484fa30f081696d308b8)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:38.759175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:38.809162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cad24748e280484fa30f081696d308b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:38.816882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cad24748e280484fa30f081696d308b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:38.817645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cad24748e280484fa30f081696d308b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:38.818369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cad24748e280484fa30f081696d308b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:38.826113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cad24748e280484fa30f081696d308b8'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:38.841015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:38.847159 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1241/2499] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:58:38 2022] DELETE /identity/v3/users/cad24748e280484fa30f081696d308b8 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:38.852546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eb5c3d-74b9-4761-b73c-0e88ceaf1505 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:38.852875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eb5c3d-74b9-4761-b73c-0e88ceaf1505 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:38.853160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eb5c3d-74b9-4761-b73c-0e88ceaf1505 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:38.866845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:38.873528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-02eb5c3d-74b9-4761-b73c-0e88ceaf1505 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:38.874759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-02eb5c3d-74b9-4761-b73c-0e88ceaf1505 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:38.934753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1259/2500] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:38.944304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:38.945811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:39.004401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d688b45f-82f7-4bde-bc55-3d169e3f5f8a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:39.015633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463f37a9-a74e-4220-9ba2-4a7a7d531248 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:39.016374 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463f37a9-a74e-4220-9ba2-4a7a7d531248 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:39.016705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463f37a9-a74e-4220-9ba2-4a7a7d531248 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:39.017006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463f37a9-a74e-4220-9ba2-4a7a7d531248 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:39.027730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463f37a9-a74e-4220-9ba2-4a7a7d531248 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:39.029222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463f37a9-a74e-4220-9ba2-4a7a7d531248 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:39.041373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1242/2501] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:39.332432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eb5c3d-74b9-4761-b73c-0e88ceaf1505 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:39.333967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-02eb5c3d-74b9-4761-b73c-0e88ceaf1505 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:39.351731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eb5c3d-74b9-4761-b73c-0e88ceaf1505 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:39.356839 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:39.357640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:39.357827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:39.358142 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] PATH_INFO: `/v3/projects/aedc46077c3d41ee9f7f9e80a8af9f27` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:39.359413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aedc46077c3d41ee9f7f9e80a8af9f27)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:39.361235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:39.418922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aedc46077c3d41ee9f7f9e80a8af9f27', 'request_id': 'req-b824a5c9-ffac-457e-82a4-8ad035cc64f9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:39.425250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aedc46077c3d41ee9f7f9e80a8af9f27', 'request_id': 'req-b824a5c9-ffac-457e-82a4-8ad035cc64f9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:39.434404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aedc46077c3d41ee9f7f9e80a8af9f27', 'request_id': 'req-b824a5c9-ffac-457e-82a4-8ad035cc64f9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:39.446797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] The token cache is being invalidate because project aedc46077c3d41ee9f7f9e80a8af9f27 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:58:39.447195 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:39.447939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:39.449592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1260/2502] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:58:39 2022] DELETE /identity/v3/projects/aedc46077c3d41ee9f7f9e80a8af9f27 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:39.453823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463f37a9-a74e-4220-9ba2-4a7a7d531248 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:39.464765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463f37a9-a74e-4220-9ba2-4a7a7d531248 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:39.527601 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:39.528261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:39.528548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:39.528863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] PATH_INFO: `/v3/users/b09ff3085e7a4008af47ba98b4179b7b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:39.530316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b09ff3085e7a4008af47ba98b4179b7b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:39.533742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:39.583587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b09ff3085e7a4008af47ba98b4179b7b'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:39.591029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b09ff3085e7a4008af47ba98b4179b7b'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:39.591639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b09ff3085e7a4008af47ba98b4179b7b'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:39.592232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b09ff3085e7a4008af47ba98b4179b7b'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:39.599101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-e0b59d77-d3be-4b0f-aeb9-89ca2ef564c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b09ff3085e7a4008af47ba98b4179b7b'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:39.618762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1243/2503] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:58:39 2022] DELETE /identity/v3/users/b09ff3085e7a4008af47ba98b4179b7b => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:39.904997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:39.906667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:39.923394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b824a5c9-ffac-457e-82a4-8ad035cc64f9 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:39.989328 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:39.990038 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:39.990343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:39.990591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] PATH_INFO: `/v3/projects/50af6523a3b04b8a962964f59e890d57` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:39.992654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] RBAC: Authorizing `identity:delete_project(project_id=50af6523a3b04b8a962964f59e890d57)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:39.994632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:40.034287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:40.049773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50af6523a3b04b8a962964f59e890d57', 'request_id': 'req-3fefd5cd-d10a-4207-8274-85d2ba1a601a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:40.058412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50af6523a3b04b8a962964f59e890d57', 'request_id': 'req-3fefd5cd-d10a-4207-8274-85d2ba1a601a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:40.067179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50af6523a3b04b8a962964f59e890d57', 'request_id': 'req-3fefd5cd-d10a-4207-8274-85d2ba1a601a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:40.080721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] The token cache is being invalidate because project 50af6523a3b04b8a962964f59e890d57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:58:40.081021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:40.081751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3fefd5cd-d10a-4207-8274-85d2ba1a601a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:40.083317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1261/2504] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:58:39 2022] DELETE /identity/v3/projects/50af6523a3b04b8a962964f59e890d57 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:40.089261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ce5135-8d70-4a5c-a0df-9d896b7d02bd None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.089716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ce5135-8d70-4a5c-a0df-9d896b7d02bd None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.090081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ce5135-8d70-4a5c-a0df-9d896b7d02bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.115656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-10ce5135-8d70-4a5c-a0df-9d896b7d02bd None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:40.117998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-10ce5135-8d70-4a5c-a0df-9d896b7d02bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:40.134702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:40.188353 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1244/2505] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:40.196230 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bb62c4-5af2-4aa0-b27f-9f4e05c9c0f6 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.196553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bb62c4-5af2-4aa0-b27f-9f4e05c9c0f6 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.196891 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bb62c4-5af2-4aa0-b27f-9f4e05c9c0f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.217702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-37bb62c4-5af2-4aa0-b27f-9f4e05c9c0f6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:40.219240 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-37bb62c4-5af2-4aa0-b27f-9f4e05c9c0f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:40.228182 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1262/2506] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:40.232716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ce5135-8d70-4a5c-a0df-9d896b7d02bd None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:40.242705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ce5135-8d70-4a5c-a0df-9d896b7d02bd None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:40.247458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50ed063-9677-48f0-82d0-9e428686c0d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:40.248071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50ed063-9677-48f0-82d0-9e428686c0d7 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.248362 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50ed063-9677-48f0-82d0-9e428686c0d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.248677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50ed063-9677-48f0-82d0-9e428686c0d7 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.249385 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50ed063-9677-48f0-82d0-9e428686c0d7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:58:40.249693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50ed063-9677-48f0-82d0-9e428686c0d7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:40.251324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50ed063-9677-48f0-82d0-9e428686c0d7 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:40.262750 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1245/2507] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:58:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 04:58:40.266956 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bb62c4-5af2-4aa0-b27f-9f4e05c9c0f6 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:40.276521 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bb62c4-5af2-4aa0-b27f-9f4e05c9c0f6 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:40.281149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8b9331-f57a-40b0-8491-1d2bceada12d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:40.281749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8b9331-f57a-40b0-8491-1d2bceada12d admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.282032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8b9331-f57a-40b0-8491-1d2bceada12d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.282308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8b9331-f57a-40b0-8491-1d2bceada12d admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.282963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8b9331-f57a-40b0-8491-1d2bceada12d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:40.284930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8b9331-f57a-40b0-8491-1d2bceada12d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:40.312261 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1263/2508] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:58:40 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:58:40.316410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50ed063-9677-48f0-82d0-9e428686c0d7 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:40.326711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50ed063-9677-48f0-82d0-9e428686c0d7 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:40.383076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e6ca06-d071-444f-a7a0-730e4bf1b75d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:40.383623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e6ca06-d071-444f-a7a0-730e4bf1b75d admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.383913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e6ca06-d071-444f-a7a0-730e4bf1b75d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.384196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e6ca06-d071-444f-a7a0-730e4bf1b75d admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.384899 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e6ca06-d071-444f-a7a0-730e4bf1b75d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:40.386825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e6ca06-d071-444f-a7a0-730e4bf1b75d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:40.408437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1246/2509] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:58:40 2022] POST /identity/v3/users => generated 292 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:58:40.412722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8b9331-f57a-40b0-8491-1d2bceada12d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:40.424010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8b9331-f57a-40b0-8491-1d2bceada12d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:40.428594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fa2b82-0ecc-4048-a448-4e49099034c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:40.429300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa2b82-0ecc-4048-a448-4e49099034c1 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.429587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa2b82-0ecc-4048-a448-4e49099034c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.429868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa2b82-0ecc-4048-a448-4e49099034c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.430583 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fa2b82-0ecc-4048-a448-4e49099034c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:40.432165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fa2b82-0ecc-4048-a448-4e49099034c1 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:40.442072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1264/2510] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:40.445883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e6ca06-d071-444f-a7a0-730e4bf1b75d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:40.455863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e6ca06-d071-444f-a7a0-730e4bf1b75d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:40.461048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264b11c4-660f-4382-9982-94d7031abdff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:40.461655 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264b11c4-660f-4382-9982-94d7031abdff admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.461945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264b11c4-660f-4382-9982-94d7031abdff admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.462225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264b11c4-660f-4382-9982-94d7031abdff admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.462926 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264b11c4-660f-4382-9982-94d7031abdff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:40.464521 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264b11c4-660f-4382-9982-94d7031abdff admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:40.475114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1247/2511] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:40.479763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fa2b82-0ecc-4048-a448-4e49099034c1 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:40.489672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fa2b82-0ecc-4048-a448-4e49099034c1 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:40.494777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da7bcd8-b691-4a0c-b499-34de25de16bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:40.495427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da7bcd8-b691-4a0c-b499-34de25de16bf admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.495709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da7bcd8-b691-4a0c-b499-34de25de16bf admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.495984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da7bcd8-b691-4a0c-b499-34de25de16bf admin admin] PATH_INFO: `/v3/projects/ab341ade317d4ab98bd72670250dc40d/users/786190d8708b45df81a68386d148289c/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.510504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da7bcd8-b691-4a0c-b499-34de25de16bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab341ade317d4ab98bd72670250dc40d, user_id=786190d8708b45df81a68386d148289c, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:40.513189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da7bcd8-b691-4a0c-b499-34de25de16bf admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:40.527580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1265/2512] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:58:40 2022] PUT /identity/v3/projects/ab341ade317d4ab98bd72670250dc40d/users/786190d8708b45df81a68386d148289c/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:58:40.537753 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e878afe-2a3b-4c4f-9969-6eea2b5ee7a7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.538071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e878afe-2a3b-4c4f-9969-6eea2b5ee7a7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.538951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e878afe-2a3b-4c4f-9969-6eea2b5ee7a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.566971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0e878afe-2a3b-4c4f-9969-6eea2b5ee7a7 None None] MFA Rules not processed for user `786190d8708b45df81a68386d148289c`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:40.568504 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0e878afe-2a3b-4c4f-9969-6eea2b5ee7a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:40.632580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1248/2513] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:40.643410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da7bcd8-b691-4a0c-b499-34de25de16bf admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:40.645268 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5da7bcd8-b691-4a0c-b499-34de25de16bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:40.702389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da7bcd8-b691-4a0c-b499-34de25de16bf admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:40.706129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a235e509-30f6-419c-aa50-76de43296a76 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:40.706743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a235e509-30f6-419c-aa50-76de43296a76 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.707028 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a235e509-30f6-419c-aa50-76de43296a76 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.707302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a235e509-30f6-419c-aa50-76de43296a76 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.715412 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a235e509-30f6-419c-aa50-76de43296a76 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:40.716868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a235e509-30f6-419c-aa50-76de43296a76 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:40.725385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1266/2514] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:40 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:40.745314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e878afe-2a3b-4c4f-9969-6eea2b5ee7a7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:40.762400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:40.808488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e878afe-2a3b-4c4f-9969-6eea2b5ee7a7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:40.812994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:40.813630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.813915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.814199 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] PATH_INFO: `/v3/users/786190d8708b45df81a68386d148289c` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.815359 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=786190d8708b45df81a68386d148289c)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:40.817317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:40.861841 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '786190d8708b45df81a68386d148289c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:40.870592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '786190d8708b45df81a68386d148289c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:40.871234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '786190d8708b45df81a68386d148289c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:40.871854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '786190d8708b45df81a68386d148289c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:40.878701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '786190d8708b45df81a68386d148289c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:40.896226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1249/2515] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:58:40 2022] DELETE /identity/v3/users/786190d8708b45df81a68386d148289c => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:40.900964 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec69cb2c-9532-4f45-ae42-fb6b91b732b6 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:40.901256 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec69cb2c-9532-4f45-ae42-fb6b91b732b6 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:40.901557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec69cb2c-9532-4f45-ae42-fb6b91b732b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:40.921518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-ec69cb2c-9532-4f45-ae42-fb6b91b732b6 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:40.922714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ec69cb2c-9532-4f45-ae42-fb6b91b732b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:40.972442 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:40.982973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1267/2516] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:40.992552 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:40.994008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:41.052382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb14bd7c-4376-4609-85b1-7e2b42f680ed admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:41.056027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacd844f-40a2-43da-b47d-555e7f6dd556 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:41.056666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd844f-40a2-43da-b47d-555e7f6dd556 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.056951 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd844f-40a2-43da-b47d-555e7f6dd556 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.057234 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd844f-40a2-43da-b47d-555e7f6dd556 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.066055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacd844f-40a2-43da-b47d-555e7f6dd556 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:41.067429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacd844f-40a2-43da-b47d-555e7f6dd556 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:41.077147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1250/2517] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 04:58:41.353903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec69cb2c-9532-4f45-ae42-fb6b91b732b6 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:41.355399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ec69cb2c-9532-4f45-ae42-fb6b91b732b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:41.368379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec69cb2c-9532-4f45-ae42-fb6b91b732b6 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:41.373568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:41.374215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.374504 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.374789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] PATH_INFO: `/v3/projects/ab341ade317d4ab98bd72670250dc40d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.375962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab341ade317d4ab98bd72670250dc40d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:41.377919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:41.431019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab341ade317d4ab98bd72670250dc40d', 'request_id': 'req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:41.437340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab341ade317d4ab98bd72670250dc40d', 'request_id': 'req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:41.444921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab341ade317d4ab98bd72670250dc40d', 'request_id': 'req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:41.454846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] The token cache is being invalidate because project ab341ade317d4ab98bd72670250dc40d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:58:41.455136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:41.455754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1aa91ea5-5ccd-4d72-8055-3dba9961eb70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:41.457286 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1268/2518] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:58:41 2022] DELETE /identity/v3/projects/ab341ade317d4ab98bd72670250dc40d => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:41.462154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e433dd10-d6e7-489e-8f86-9cc10e805512 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.462632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e433dd10-d6e7-489e-8f86-9cc10e805512 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.462730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e433dd10-d6e7-489e-8f86-9cc10e805512 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.482256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e433dd10-d6e7-489e-8f86-9cc10e805512 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:41.483478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e433dd10-d6e7-489e-8f86-9cc10e805512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:41.516865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:41.545680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1251/2519] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:58:41.555013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b734269-8aaa-4782-98ef-d98e4c809d36 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.555303 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b734269-8aaa-4782-98ef-d98e4c809d36 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.555584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b734269-8aaa-4782-98ef-d98e4c809d36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.576092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7b734269-8aaa-4782-98ef-d98e4c809d36 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:41.577386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7b734269-8aaa-4782-98ef-d98e4c809d36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:41.586014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1269/2520] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:41.590077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e433dd10-d6e7-489e-8f86-9cc10e805512 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:41.600080 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e433dd10-d6e7-489e-8f86-9cc10e805512 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:41.604830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49119c43-60ff-497d-bc5e-25a1e3f94286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:41.605471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49119c43-60ff-497d-bc5e-25a1e3f94286 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.605758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49119c43-60ff-497d-bc5e-25a1e3f94286 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.606033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49119c43-60ff-497d-bc5e-25a1e3f94286 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.606726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49119c43-60ff-497d-bc5e-25a1e3f94286 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:58:41.607032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49119c43-60ff-497d-bc5e-25a1e3f94286 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:41.608686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49119c43-60ff-497d-bc5e-25a1e3f94286 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:41.623824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1252/2521] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:58:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:58:41.625019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b734269-8aaa-4782-98ef-d98e4c809d36 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:41.635229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b734269-8aaa-4782-98ef-d98e4c809d36 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:41.639953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38763368-c8ae-4c1f-924f-e4c38efa9287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:41.640694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38763368-c8ae-4c1f-924f-e4c38efa9287 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.641051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38763368-c8ae-4c1f-924f-e4c38efa9287 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.641400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38763368-c8ae-4c1f-924f-e4c38efa9287 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.642139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38763368-c8ae-4c1f-924f-e4c38efa9287 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:41.644093 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38763368-c8ae-4c1f-924f-e4c38efa9287 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:41.675068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1270/2522] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:58:41 2022] POST /identity/v3/projects => generated 385 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:58:41.679772 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49119c43-60ff-497d-bc5e-25a1e3f94286 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:41.689706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49119c43-60ff-497d-bc5e-25a1e3f94286 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:41.747201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d79f5c2-6d89-4454-a370-9f2d67e61e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:41.747824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d79f5c2-6d89-4454-a370-9f2d67e61e4d admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.748111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d79f5c2-6d89-4454-a370-9f2d67e61e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.748392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d79f5c2-6d89-4454-a370-9f2d67e61e4d admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.749103 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d79f5c2-6d89-4454-a370-9f2d67e61e4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:41.751019 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d79f5c2-6d89-4454-a370-9f2d67e61e4d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:41.772222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1253/2523] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:58:41 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:58:41.776420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38763368-c8ae-4c1f-924f-e4c38efa9287 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:41.786592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38763368-c8ae-4c1f-924f-e4c38efa9287 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:41.791705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b79dcae-06cf-4174-ab31-d3a515ce2564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:41.792311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b79dcae-06cf-4174-ab31-d3a515ce2564 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.792616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b79dcae-06cf-4174-ab31-d3a515ce2564 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.792911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b79dcae-06cf-4174-ab31-d3a515ce2564 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.793605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b79dcae-06cf-4174-ab31-d3a515ce2564 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:41.795200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b79dcae-06cf-4174-ab31-d3a515ce2564 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:41.806377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1271/2524] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:41.810300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d79f5c2-6d89-4454-a370-9f2d67e61e4d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:41.820945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d79f5c2-6d89-4454-a370-9f2d67e61e4d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:41.825610 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c327ef1-2723-49b0-888d-84513bca4aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:41.826567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c327ef1-2723-49b0-888d-84513bca4aed admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.826567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c327ef1-2723-49b0-888d-84513bca4aed admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.826785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c327ef1-2723-49b0-888d-84513bca4aed admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.827482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c327ef1-2723-49b0-888d-84513bca4aed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:41.829137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c327ef1-2723-49b0-888d-84513bca4aed admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:41.839859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1254/2525] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:41.843913 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b79dcae-06cf-4174-ab31-d3a515ce2564 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:41.854383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b79dcae-06cf-4174-ab31-d3a515ce2564 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:41.858998 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af9eaea-bfaa-41d0-85e5-bd5de1ed43c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:41.859653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af9eaea-bfaa-41d0-85e5-bd5de1ed43c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.859939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af9eaea-bfaa-41d0-85e5-bd5de1ed43c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.860216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af9eaea-bfaa-41d0-85e5-bd5de1ed43c9 admin admin] PATH_INFO: `/v3/projects/159e3bc8e2c24b039e29a02ce820014b/users/2886ad033d134397815e14210035b027/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.876546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af9eaea-bfaa-41d0-85e5-bd5de1ed43c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=159e3bc8e2c24b039e29a02ce820014b, user_id=2886ad033d134397815e14210035b027, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:41.879501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af9eaea-bfaa-41d0-85e5-bd5de1ed43c9 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:41.895281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1272/2526] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:58:41 2022] PUT /identity/v3/projects/159e3bc8e2c24b039e29a02ce820014b/users/2886ad033d134397815e14210035b027/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:58:41.905691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b335dcb-a0fd-41ef-b94e-7e0693a0c165 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:41.905986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b335dcb-a0fd-41ef-b94e-7e0693a0c165 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:41.906292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b335dcb-a0fd-41ef-b94e-7e0693a0c165 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:41.935105 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0b335dcb-a0fd-41ef-b94e-7e0693a0c165 None None] MFA Rules not processed for user `2886ad033d134397815e14210035b027`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:41.936343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0b335dcb-a0fd-41ef-b94e-7e0693a0c165 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:41.999947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1255/2527] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:41 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:42.011388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af9eaea-bfaa-41d0-85e5-bd5de1ed43c9 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:42.013158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-1af9eaea-bfaa-41d0-85e5-bd5de1ed43c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:42.070482 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af9eaea-bfaa-41d0-85e5-bd5de1ed43c9 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:42.074183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969d719b-6892-4259-b58d-c1e4a8671659 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:42.074801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969d719b-6892-4259-b58d-c1e4a8671659 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:42.075089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969d719b-6892-4259-b58d-c1e4a8671659 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:42.075365 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969d719b-6892-4259-b58d-c1e4a8671659 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:42.083498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969d719b-6892-4259-b58d-c1e4a8671659 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:42.085658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-969d719b-6892-4259-b58d-c1e4a8671659 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:42.094753 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1273/2528] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:42 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:51.997037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=82,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:52.090432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=80,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:52.805206 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b335dcb-a0fd-41ef-b94e-7e0693a0c165 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:52.814976 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b335dcb-a0fd-41ef-b94e-7e0693a0c165 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:52.818477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fa2947-4074-438d-9493-99ae07940fcc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:52.819093 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fa2947-4074-438d-9493-99ae07940fcc service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:52.819383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fa2947-4074-438d-9493-99ae07940fcc service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:52.819666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fa2947-4074-438d-9493-99ae07940fcc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:52.821978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-88fa2947-4074-438d-9493-99ae07940fcc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:52.902692 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fa2947-4074-438d-9493-99ae07940fcc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:52.904157 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fa2947-4074-438d-9493-99ae07940fcc service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:52.922442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1256/2529] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:52 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:55.744104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969d719b-6892-4259-b58d-c1e4a8671659 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:55.821247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969d719b-6892-4259-b58d-c1e4a8671659 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:55.825782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:55.826425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:55.826711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:55.826990 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] PATH_INFO: `/v3/users/2886ad033d134397815e14210035b027` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:55.828143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2886ad033d134397815e14210035b027)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:55.830106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:55.878025 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2886ad033d134397815e14210035b027'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:55.878671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2886ad033d134397815e14210035b027'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:55.879279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2886ad033d134397815e14210035b027'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:55.891305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2886ad033d134397815e14210035b027'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:55.898026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2886ad033d134397815e14210035b027'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:55.911764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1274/2530] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:58:55 2022] DELETE /identity/v3/users/2886ad033d134397815e14210035b027 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:55.917680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a6486-e4a9-41a8-9a64-918a397e7f3e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:55.918184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a6486-e4a9-41a8-9a64-918a397e7f3e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:55.918291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373a6486-e4a9-41a8-9a64-918a397e7f3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:55.937098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-373a6486-e4a9-41a8-9a64-918a397e7f3e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:55.938312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-373a6486-e4a9-41a8-9a64-918a397e7f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:55.997009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1257/2531] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:56.006743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:56.008243 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:56.067493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b094d3-e13c-4855-abea-61ab2adb2713 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:56.071892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ef1f65-df28-4276-bf75-e01c91c97f9a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:56.072513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ef1f65-df28-4276-bf75-e01c91c97f9a service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.072845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ef1f65-df28-4276-bf75-e01c91c97f9a service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.073127 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ef1f65-df28-4276-bf75-e01c91c97f9a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.082206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ef1f65-df28-4276-bf75-e01c91c97f9a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:56.083643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ef1f65-df28-4276-bf75-e01c91c97f9a service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:56.093255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1275/2532] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:58:56.380806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a6486-e4a9-41a8-9a64-918a397e7f3e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:56.382793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-373a6486-e4a9-41a8-9a64-918a397e7f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:56.407406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373a6486-e4a9-41a8-9a64-918a397e7f3e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:56.413153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:56.413938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.414875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.415161 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] PATH_INFO: `/v3/projects/159e3bc8e2c24b039e29a02ce820014b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.416338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=159e3bc8e2c24b039e29a02ce820014b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:56.418600 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:56.465980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '159e3bc8e2c24b039e29a02ce820014b', 'request_id': 'req-279be5d2-536d-4d9c-a478-9ec8eebb9e06'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:56.473096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '159e3bc8e2c24b039e29a02ce820014b', 'request_id': 'req-279be5d2-536d-4d9c-a478-9ec8eebb9e06'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:56.479360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '159e3bc8e2c24b039e29a02ce820014b', 'request_id': 'req-279be5d2-536d-4d9c-a478-9ec8eebb9e06'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:56.481089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:56.495639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] The token cache is being invalidate because project 159e3bc8e2c24b039e29a02ce820014b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:58:56.495929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:56.496576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-279be5d2-536d-4d9c-a478-9ec8eebb9e06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:58:56.498137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1258/2533] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:58:56 2022] DELETE /identity/v3/projects/159e3bc8e2c24b039e29a02ce820014b => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:56.502995 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d999ed-7098-4bef-b94e-2b7b04ffdccf None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.503294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d999ed-7098-4bef-b94e-2b7b04ffdccf None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.503579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d999ed-7098-4bef-b94e-2b7b04ffdccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.522929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-75d999ed-7098-4bef-b94e-2b7b04ffdccf None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:56.524801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-75d999ed-7098-4bef-b94e-2b7b04ffdccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:56.542282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:58:56.587345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1276/2534] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:58:56.596704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5ba1e5-d61e-4aca-a42e-442334a3922d None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.597131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5ba1e5-d61e-4aca-a42e-442334a3922d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.597485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5ba1e5-d61e-4aca-a42e-442334a3922d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.619559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6b5ba1e5-d61e-4aca-a42e-442334a3922d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:56.621732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6b5ba1e5-d61e-4aca-a42e-442334a3922d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:56.631432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1259/2535] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:58:56.636034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d999ed-7098-4bef-b94e-2b7b04ffdccf None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:56.650294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d999ed-7098-4bef-b94e-2b7b04ffdccf None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:56.657773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dae4961-e266-4dba-8c76-5134dd2d3993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:56.658423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dae4961-e266-4dba-8c76-5134dd2d3993 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.658942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dae4961-e266-4dba-8c76-5134dd2d3993 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.659457 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dae4961-e266-4dba-8c76-5134dd2d3993 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.660763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dae4961-e266-4dba-8c76-5134dd2d3993 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:58:56.661085 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dae4961-e266-4dba-8c76-5134dd2d3993 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:56.663378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dae4961-e266-4dba-8c76-5134dd2d3993 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:56.679905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1277/2536] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:58:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:58:56.684323 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5ba1e5-d61e-4aca-a42e-442334a3922d None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:56.694037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5ba1e5-d61e-4aca-a42e-442334a3922d None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:56.698736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31f3962-1eda-47b7-b566-bb4c092c109c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:56.699355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31f3962-1eda-47b7-b566-bb4c092c109c admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.699641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31f3962-1eda-47b7-b566-bb4c092c109c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.699920 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31f3962-1eda-47b7-b566-bb4c092c109c admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.700623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31f3962-1eda-47b7-b566-bb4c092c109c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:56.702595 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31f3962-1eda-47b7-b566-bb4c092c109c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:56.731009 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1260/2537] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:58:56 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:58:56.735007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dae4961-e266-4dba-8c76-5134dd2d3993 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:56.744926 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dae4961-e266-4dba-8c76-5134dd2d3993 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:56.801701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7697b5-380e-49a1-b214-44df1cb998da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:56.802316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7697b5-380e-49a1-b214-44df1cb998da admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.802623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7697b5-380e-49a1-b214-44df1cb998da admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.802899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7697b5-380e-49a1-b214-44df1cb998da admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.803556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7697b5-380e-49a1-b214-44df1cb998da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:56.805607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e7697b5-380e-49a1-b214-44df1cb998da admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:56.826373 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1278/2538] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:58:56 2022] POST /identity/v3/users => generated 288 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:58:56.830712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31f3962-1eda-47b7-b566-bb4c092c109c admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:56.841842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31f3962-1eda-47b7-b566-bb4c092c109c admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:56.846483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832fef7f-a664-4728-9d23-0a1fd731c4e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:56.847094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832fef7f-a664-4728-9d23-0a1fd731c4e9 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.847383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832fef7f-a664-4728-9d23-0a1fd731c4e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.847664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832fef7f-a664-4728-9d23-0a1fd731c4e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.848362 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832fef7f-a664-4728-9d23-0a1fd731c4e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:56.850112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-832fef7f-a664-4728-9d23-0a1fd731c4e9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:56.860431 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1261/2539] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:56 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:56.864217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7697b5-380e-49a1-b214-44df1cb998da admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:56.874934 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7697b5-380e-49a1-b214-44df1cb998da admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:56.879652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4c328b-702f-4210-b89d-867375a314f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:56.880263 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4c328b-702f-4210-b89d-867375a314f8 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.880553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4c328b-702f-4210-b89d-867375a314f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.880874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4c328b-702f-4210-b89d-867375a314f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.881571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4c328b-702f-4210-b89d-867375a314f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:56.883819 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4c328b-702f-4210-b89d-867375a314f8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:56.894443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1279/2540] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:58:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:58:56.898556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832fef7f-a664-4728-9d23-0a1fd731c4e9 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:56.908452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832fef7f-a664-4728-9d23-0a1fd731c4e9 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:56.913246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a14335-ea9f-4682-b948-2b5e985ff68c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:56.913905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a14335-ea9f-4682-b948-2b5e985ff68c admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.914239 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a14335-ea9f-4682-b948-2b5e985ff68c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.914514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a14335-ea9f-4682-b948-2b5e985ff68c admin admin] PATH_INFO: `/v3/projects/7b65d8b33aac4563b7f70ecb8d66cca6/users/7d9949cf0c7d4a2bb6ae5802135bbc8d/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.931611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a14335-ea9f-4682-b948-2b5e985ff68c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7b65d8b33aac4563b7f70ecb8d66cca6, user_id=7d9949cf0c7d4a2bb6ae5802135bbc8d, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:56.934370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a14335-ea9f-4682-b948-2b5e985ff68c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:56.947878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1262/2541] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:58:56 2022] PUT /identity/v3/projects/7b65d8b33aac4563b7f70ecb8d66cca6/users/7d9949cf0c7d4a2bb6ae5802135bbc8d/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:58:56.958141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722949a2-f9a8-4bbd-b6bc-90c54d2b64d0 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:56.958429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722949a2-f9a8-4bbd-b6bc-90c54d2b64d0 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:56.958702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722949a2-f9a8-4bbd-b6bc-90c54d2b64d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:56.985688 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-722949a2-f9a8-4bbd-b6bc-90c54d2b64d0 None None] MFA Rules not processed for user `7d9949cf0c7d4a2bb6ae5802135bbc8d`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:58:56.986804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-722949a2-f9a8-4bbd-b6bc-90c54d2b64d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:57.050915 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1280/2542] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:58:56 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:58:57.061982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a14335-ea9f-4682-b948-2b5e985ff68c admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:58:57.063710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-19a14335-ea9f-4682-b948-2b5e985ff68c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:58:57.122993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a14335-ea9f-4682-b948-2b5e985ff68c admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:58:57.126683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89658e9-0b5f-4236-aa59-2aa7b0bfc6d3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:58:57.127302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89658e9-0b5f-4236-aa59-2aa7b0bfc6d3 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:58:57.127589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89658e9-0b5f-4236-aa59-2aa7b0bfc6d3 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:58:57.127870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89658e9-0b5f-4236-aa59-2aa7b0bfc6d3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:58:57.136571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89658e9-0b5f-4236-aa59-2aa7b0bfc6d3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:58:57.138094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89658e9-0b5f-4236-aa59-2aa7b0bfc6d3 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:58:57.146865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1263/2543] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:58:57 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:04.413340 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722949a2-f9a8-4bbd-b6bc-90c54d2b64d0 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:04.436083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:04.484044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722949a2-f9a8-4bbd-b6bc-90c54d2b64d0 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:04.488657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:04.489309 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:04.489601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:04.489880 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] PATH_INFO: `/v3/users/7d9949cf0c7d4a2bb6ae5802135bbc8d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:04.491099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d9949cf0c7d4a2bb6ae5802135bbc8d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:04.494148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:04.540632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d9949cf0c7d4a2bb6ae5802135bbc8d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:04.541209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d9949cf0c7d4a2bb6ae5802135bbc8d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:04.542072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d9949cf0c7d4a2bb6ae5802135bbc8d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:04.553564 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d9949cf0c7d4a2bb6ae5802135bbc8d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:04.560834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d9949cf0c7d4a2bb6ae5802135bbc8d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:04.573234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1281/2544] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:59:04 2022] DELETE /identity/v3/users/7d9949cf0c7d4a2bb6ae5802135bbc8d => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:04.578028 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1308b1-f11b-4f2f-af0c-c5dca19c5af9 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:04.578512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1308b1-f11b-4f2f-af0c-c5dca19c5af9 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:04.578625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff1308b1-f11b-4f2f-af0c-c5dca19c5af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:04.597206 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-ff1308b1-f11b-4f2f-af0c-c5dca19c5af9 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:04.598401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ff1308b1-f11b-4f2f-af0c-c5dca19c5af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:04.656722 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1264/2545] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:04.667411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:04.668976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:04.725487 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf9cf6f-d40a-47b9-ab51-7ce7fd7a40bc admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:04.729685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389868b9-adfa-4ce4-8413-5abeb3c08eb5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:04.730302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389868b9-adfa-4ce4-8413-5abeb3c08eb5 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:04.730591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389868b9-adfa-4ce4-8413-5abeb3c08eb5 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:04.730877 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389868b9-adfa-4ce4-8413-5abeb3c08eb5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:04.739744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389868b9-adfa-4ce4-8413-5abeb3c08eb5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:04.741208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389868b9-adfa-4ce4-8413-5abeb3c08eb5 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:04.751409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1282/2546] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:05.022366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1308b1-f11b-4f2f-af0c-c5dca19c5af9 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:05.023862 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ff1308b1-f11b-4f2f-af0c-c5dca19c5af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:05.040959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff1308b1-f11b-4f2f-af0c-c5dca19c5af9 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:05.046155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:05.046802 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.047190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.047390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] PATH_INFO: `/v3/projects/7b65d8b33aac4563b7f70ecb8d66cca6` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.048560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7b65d8b33aac4563b7f70ecb8d66cca6)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:05.050519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:05.083283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:05.100581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7b65d8b33aac4563b7f70ecb8d66cca6', 'request_id': 'req-a8705444-66cc-4e29-a6c6-a504076f61f6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:05.111016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7b65d8b33aac4563b7f70ecb8d66cca6', 'request_id': 'req-a8705444-66cc-4e29-a6c6-a504076f61f6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:05.117561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7b65d8b33aac4563b7f70ecb8d66cca6', 'request_id': 'req-a8705444-66cc-4e29-a6c6-a504076f61f6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:05.129266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] The token cache is being invalidate because project 7b65d8b33aac4563b7f70ecb8d66cca6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:59:05.129559 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:05.130219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a8705444-66cc-4e29-a6c6-a504076f61f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:05.131742 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1265/2547] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:59:05 2022] DELETE /identity/v3/projects/7b65d8b33aac4563b7f70ecb8d66cca6 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:05.138231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c0af6-c58c-4b73-b546-c55bfece509d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.138646 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c0af6-c58c-4b73-b546-c55bfece509d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.139001 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c0af6-c58c-4b73-b546-c55bfece509d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.159066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-733c0af6-c58c-4b73-b546-c55bfece509d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:05.160429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-733c0af6-c58c-4b73-b546-c55bfece509d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:05.201527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:05.235004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1283/2548] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:05.243129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f1a300-808f-4da5-8da3-764d68973988 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.243498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f1a300-808f-4da5-8da3-764d68973988 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.243850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f1a300-808f-4da5-8da3-764d68973988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.265417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-e6f1a300-808f-4da5-8da3-764d68973988 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:05.266718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-e6f1a300-808f-4da5-8da3-764d68973988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:05.279249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1266/2549] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:05.283710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733c0af6-c58c-4b73-b546-c55bfece509d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:05.295275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733c0af6-c58c-4b73-b546-c55bfece509d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:05.300139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d974c78-7f1e-4daf-b1e4-da47bee3de66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:05.300796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d974c78-7f1e-4daf-b1e4-da47bee3de66 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.301083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d974c78-7f1e-4daf-b1e4-da47bee3de66 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.301358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d974c78-7f1e-4daf-b1e4-da47bee3de66 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.302079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d974c78-7f1e-4daf-b1e4-da47bee3de66 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:59:05.302369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d974c78-7f1e-4daf-b1e4-da47bee3de66 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:05.303970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d974c78-7f1e-4daf-b1e4-da47bee3de66 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:05.322958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1284/2550] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:59:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:59:05.324312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f1a300-808f-4da5-8da3-764d68973988 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:05.333942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f1a300-808f-4da5-8da3-764d68973988 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:05.350003 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81180f52-32db-4b34-819f-407e27913a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:05.351032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81180f52-32db-4b34-819f-407e27913a60 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.352017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81180f52-32db-4b34-819f-407e27913a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.352950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81180f52-32db-4b34-819f-407e27913a60 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.353642 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81180f52-32db-4b34-819f-407e27913a60 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:05.355568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81180f52-32db-4b34-819f-407e27913a60 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:05.392001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1267/2551] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:59:05 2022] POST /identity/v3/projects => generated 369 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:05.397122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d974c78-7f1e-4daf-b1e4-da47bee3de66 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:05.409681 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d974c78-7f1e-4daf-b1e4-da47bee3de66 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:05.472480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106b1ff8-cf91-4880-a515-44ab99e233b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:05.473238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106b1ff8-cf91-4880-a515-44ab99e233b5 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.473599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106b1ff8-cf91-4880-a515-44ab99e233b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.473950 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106b1ff8-cf91-4880-a515-44ab99e233b5 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.474702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106b1ff8-cf91-4880-a515-44ab99e233b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:05.477153 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106b1ff8-cf91-4880-a515-44ab99e233b5 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:05.502820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1285/2552] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:59:05 2022] POST /identity/v3/users => generated 285 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:05.506752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81180f52-32db-4b34-819f-407e27913a60 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:05.519142 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81180f52-32db-4b34-819f-407e27913a60 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:05.524767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935ca74e-b4d6-4d2f-a5eb-547d2a69d728 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:05.525479 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935ca74e-b4d6-4d2f-a5eb-547d2a69d728 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.525839 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935ca74e-b4d6-4d2f-a5eb-547d2a69d728 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.526190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935ca74e-b4d6-4d2f-a5eb-547d2a69d728 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.526965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935ca74e-b4d6-4d2f-a5eb-547d2a69d728 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:05.528933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935ca74e-b4d6-4d2f-a5eb-547d2a69d728 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:05.540921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1268/2553] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:05 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:05.545086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106b1ff8-cf91-4880-a515-44ab99e233b5 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:05.556165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106b1ff8-cf91-4880-a515-44ab99e233b5 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:05.563862 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81db16f1-a80f-4252-883b-1c942488f002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:05.564492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db16f1-a80f-4252-883b-1c942488f002 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.564813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db16f1-a80f-4252-883b-1c942488f002 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.565096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81db16f1-a80f-4252-883b-1c942488f002 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.565800 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81db16f1-a80f-4252-883b-1c942488f002 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:05.567410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81db16f1-a80f-4252-883b-1c942488f002 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:05.579776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1286/2554] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:05 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:59:05.584122 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935ca74e-b4d6-4d2f-a5eb-547d2a69d728 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:05.595796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935ca74e-b4d6-4d2f-a5eb-547d2a69d728 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:05.600796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de051b62-8dc5-40dc-8d2c-7a2a16b148f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:05.601584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de051b62-8dc5-40dc-8d2c-7a2a16b148f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.602589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de051b62-8dc5-40dc-8d2c-7a2a16b148f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.602946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de051b62-8dc5-40dc-8d2c-7a2a16b148f1 admin admin] PATH_INFO: `/v3/projects/e5a73fe26f134102bf88e5e06da0866c/users/ca007ca744e9499cb236aa4e7831d1cc/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.620028 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de051b62-8dc5-40dc-8d2c-7a2a16b148f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e5a73fe26f134102bf88e5e06da0866c, user_id=ca007ca744e9499cb236aa4e7831d1cc, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:05.623306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de051b62-8dc5-40dc-8d2c-7a2a16b148f1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:05.641452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1269/2555] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:05 2022] PUT /identity/v3/projects/e5a73fe26f134102bf88e5e06da0866c/users/ca007ca744e9499cb236aa4e7831d1cc/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:59:05.712538 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40df9a50-f002-4388-80b2-11ab30b7ffaa None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.712851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40df9a50-f002-4388-80b2-11ab30b7ffaa None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.713132 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40df9a50-f002-4388-80b2-11ab30b7ffaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.741491 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-40df9a50-f002-4388-80b2-11ab30b7ffaa None None] MFA Rules not processed for user `ca007ca744e9499cb236aa4e7831d1cc`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:05.743278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-40df9a50-f002-4388-80b2-11ab30b7ffaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:05.811066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1287/2556] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:05 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:05.823214 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de051b62-8dc5-40dc-8d2c-7a2a16b148f1 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:05.825005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-de051b62-8dc5-40dc-8d2c-7a2a16b148f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:05.926030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de051b62-8dc5-40dc-8d2c-7a2a16b148f1 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:05.930721 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09606e13-c4d3-4dc0-958a-65d795c7419a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3749dc021aac478f88072ced48f4cee4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:05.931495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09606e13-c4d3-4dc0-958a-65d795c7419a service glance] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:05.931854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09606e13-c4d3-4dc0-958a-65d795c7419a service glance] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:05.932356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09606e13-c4d3-4dc0-958a-65d795c7419a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:05.941624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09606e13-c4d3-4dc0-958a-65d795c7419a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:05.943307 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09606e13-c4d3-4dc0-958a-65d795c7419a service glance] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:05.955412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1270/2557] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:05 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:06.124854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40df9a50-f002-4388-80b2-11ab30b7ffaa None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:06.126973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-40df9a50-f002-4388-80b2-11ab30b7ffaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:06.217241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:06.292828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40df9a50-f002-4388-80b2-11ab30b7ffaa None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:06.299813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d03b847-41f1-4842-9fdc-e9fb6089bdea service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:06.300460 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03b847-41f1-4842-9fdc-e9fb6089bdea service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:06.300813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03b847-41f1-4842-9fdc-e9fb6089bdea service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:06.301103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d03b847-41f1-4842-9fdc-e9fb6089bdea service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:06.302971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5d03b847-41f1-4842-9fdc-e9fb6089bdea service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:06.327050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d03b847-41f1-4842-9fdc-e9fb6089bdea service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:06.329345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d03b847-41f1-4842-9fdc-e9fb6089bdea service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:06.353618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1288/2558] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:06 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:15.949799 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=76,keystone:DELETE=5,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:16.339548 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=18 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:22.767052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09606e13-c4d3-4dc0-958a-65d795c7419a service glance] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:22.768707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-09606e13-c4d3-4dc0-958a-65d795c7419a service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:22.826749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09606e13-c4d3-4dc0-958a-65d795c7419a service glance] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:22.830430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a26ff37-b45a-45c1-ac57-c5a774ec4551 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:22.831043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26ff37-b45a-45c1-ac57-c5a774ec4551 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:22.831327 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26ff37-b45a-45c1-ac57-c5a774ec4551 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:22.831605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26ff37-b45a-45c1-ac57-c5a774ec4551 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:22.833381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7a26ff37-b45a-45c1-ac57-c5a774ec4551 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:22.897554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a26ff37-b45a-45c1-ac57-c5a774ec4551 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:22.898987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a26ff37-b45a-45c1-ac57-c5a774ec4551 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:22.908584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1271/2559] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:26.087149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d03b847-41f1-4842-9fdc-e9fb6089bdea service nova] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:26.090752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5d03b847-41f1-4842-9fdc-e9fb6089bdea service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:26.107735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d03b847-41f1-4842-9fdc-e9fb6089bdea service nova] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:26.112435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:26.113120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.113411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.113697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] PATH_INFO: `/v3/users/572408fd157844dcae13000d8e9c4b7a` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.114882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] RBAC: Authorizing `identity:delete_user(user_id=572408fd157844dcae13000d8e9c4b7a)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:26.116919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:26.164177 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '572408fd157844dcae13000d8e9c4b7a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:26.164864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '572408fd157844dcae13000d8e9c4b7a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:26.166204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '572408fd157844dcae13000d8e9c4b7a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:26.178409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '572408fd157844dcae13000d8e9c4b7a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:26.185483 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-54dab25a-c0e3-4684-8ea5-4564fa35e77f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '572408fd157844dcae13000d8e9c4b7a'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:26.197802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1289/2560] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:59:26 2022] DELETE /identity/v3/users/572408fd157844dcae13000d8e9c4b7a => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:26.378326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a26ff37-b45a-45c1-ac57-c5a774ec4551 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:26.389875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a26ff37-b45a-45c1-ac57-c5a774ec4551 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:26.449174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:26.449822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.450114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.450398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] PATH_INFO: `/v3/projects/461e2af738084ccb986169d752510588` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.451561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] RBAC: Authorizing `identity:delete_project(project_id=461e2af738084ccb986169d752510588)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:26.453546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:26.500567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '461e2af738084ccb986169d752510588', 'request_id': 'req-6d878266-dd57-4f4a-9e94-f085fb04a373'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:26.502973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:26.507985 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '461e2af738084ccb986169d752510588', 'request_id': 'req-6d878266-dd57-4f4a-9e94-f085fb04a373'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:26.515624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '461e2af738084ccb986169d752510588', 'request_id': 'req-6d878266-dd57-4f4a-9e94-f085fb04a373'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:26.527086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] The token cache is being invalidate because project 461e2af738084ccb986169d752510588 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:59:26.527434 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:26.528077 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-6d878266-dd57-4f4a-9e94-f085fb04a373 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:26.529610 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1272/2561] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:59:26 2022] DELETE /identity/v3/projects/461e2af738084ccb986169d752510588 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:59:26.534616 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2155e640-7be9-49aa-b8bc-e768ea690215 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.534919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2155e640-7be9-49aa-b8bc-e768ea690215 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.535205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2155e640-7be9-49aa-b8bc-e768ea690215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.555163 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2155e640-7be9-49aa-b8bc-e768ea690215 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:26.556496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2155e640-7be9-49aa-b8bc-e768ea690215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:26.616861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1290/2562] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:26.625022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79783cdc-8b2c-41af-a4f9-d6c890af1ebf None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.625310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79783cdc-8b2c-41af-a4f9-d6c890af1ebf None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.625592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79783cdc-8b2c-41af-a4f9-d6c890af1ebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.645610 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-79783cdc-8b2c-41af-a4f9-d6c890af1ebf None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:26.646810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-79783cdc-8b2c-41af-a4f9-d6c890af1ebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:26.654638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1273/2563] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:26.658849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2155e640-7be9-49aa-b8bc-e768ea690215 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:26.669073 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2155e640-7be9-49aa-b8bc-e768ea690215 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:26.673648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1a79cd-a12f-411b-8d85-a4ac1f92734a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:26.674257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1a79cd-a12f-411b-8d85-a4ac1f92734a admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.674547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1a79cd-a12f-411b-8d85-a4ac1f92734a admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.674821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1a79cd-a12f-411b-8d85-a4ac1f92734a admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.675521 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1a79cd-a12f-411b-8d85-a4ac1f92734a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:59:26.675827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1a79cd-a12f-411b-8d85-a4ac1f92734a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:26.677459 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1a79cd-a12f-411b-8d85-a4ac1f92734a admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:26.689672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1291/2564] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:59:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:59:26.693932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79783cdc-8b2c-41af-a4f9-d6c890af1ebf None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:26.703819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79783cdc-8b2c-41af-a4f9-d6c890af1ebf None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:26.708443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9445ef8a-08cd-40ca-9dd1-51bae23ec83f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:26.709078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9445ef8a-08cd-40ca-9dd1-51bae23ec83f admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.709361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9445ef8a-08cd-40ca-9dd1-51bae23ec83f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.709635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9445ef8a-08cd-40ca-9dd1-51bae23ec83f admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.710294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9445ef8a-08cd-40ca-9dd1-51bae23ec83f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:26.712204 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9445ef8a-08cd-40ca-9dd1-51bae23ec83f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:26.739499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1274/2565] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:59:26 2022] POST /identity/v3/projects => generated 365 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:59:26.743625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1a79cd-a12f-411b-8d85-a4ac1f92734a admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:26.753936 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1a79cd-a12f-411b-8d85-a4ac1f92734a admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:26.811036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3de8647-db24-4850-a680-32a489515279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:26.811658 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3de8647-db24-4850-a680-32a489515279 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.811941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3de8647-db24-4850-a680-32a489515279 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.812220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3de8647-db24-4850-a680-32a489515279 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.812920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3de8647-db24-4850-a680-32a489515279 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:26.814852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3de8647-db24-4850-a680-32a489515279 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:26.822607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:26.837211 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1292/2566] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:59:26 2022] POST /identity/v3/users => generated 283 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:59:26.841929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9445ef8a-08cd-40ca-9dd1-51bae23ec83f admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:26.852107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9445ef8a-08cd-40ca-9dd1-51bae23ec83f admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:26.857413 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2cfc9f-e126-4ca8-a957-8640d291ca2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:26.858018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2cfc9f-e126-4ca8-a957-8640d291ca2b admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.858301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2cfc9f-e126-4ca8-a957-8640d291ca2b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.858578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2cfc9f-e126-4ca8-a957-8640d291ca2b admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.859270 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2cfc9f-e126-4ca8-a957-8640d291ca2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:26.860913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2cfc9f-e126-4ca8-a957-8640d291ca2b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:26.870843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1275/2567] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:26 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:26.874844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3de8647-db24-4850-a680-32a489515279 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:26.884666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3de8647-db24-4850-a680-32a489515279 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:26.889337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3928f0-ee6c-47a8-a446-a1c61007115e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:26.889944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3928f0-ee6c-47a8-a446-a1c61007115e admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.890232 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3928f0-ee6c-47a8-a446-a1c61007115e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.890514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3928f0-ee6c-47a8-a446-a1c61007115e admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.891217 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3928f0-ee6c-47a8-a446-a1c61007115e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:26.892875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3928f0-ee6c-47a8-a446-a1c61007115e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:26.903875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1293/2568] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:26 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:26.907775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2cfc9f-e126-4ca8-a957-8640d291ca2b admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:26.918601 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2cfc9f-e126-4ca8-a957-8640d291ca2b admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:26.923199 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a57da-88da-40af-a409-6f212bbc0bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:26.923847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a57da-88da-40af-a409-6f212bbc0bf0 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.924130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a57da-88da-40af-a409-6f212bbc0bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.924405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a57da-88da-40af-a409-6f212bbc0bf0 admin admin] PATH_INFO: `/v3/projects/e6aaf796c5db4f1ebbcb32e1b1c9010c/users/b2e058b9d30041e882040e8433b1f10d/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.938730 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a57da-88da-40af-a409-6f212bbc0bf0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6aaf796c5db4f1ebbcb32e1b1c9010c, user_id=b2e058b9d30041e882040e8433b1f10d, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:26.941406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a57da-88da-40af-a409-6f212bbc0bf0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:26.954277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1276/2569] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:26 2022] PUT /identity/v3/projects/e6aaf796c5db4f1ebbcb32e1b1c9010c/users/b2e058b9d30041e882040e8433b1f10d/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:26.964931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7caab5d-d225-4f51-a4d0-8f99410de809 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:26.965232 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7caab5d-d225-4f51-a4d0-8f99410de809 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:26.965515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7caab5d-d225-4f51-a4d0-8f99410de809 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:26.993248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c7caab5d-d225-4f51-a4d0-8f99410de809 None None] MFA Rules not processed for user `b2e058b9d30041e882040e8433b1f10d`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:26.994577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c7caab5d-d225-4f51-a4d0-8f99410de809 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:27.058203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1294/2570] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:26 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:59:27.062555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a57da-88da-40af-a409-6f212bbc0bf0 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:27.122450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a57da-88da-40af-a409-6f212bbc0bf0 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:27.127043 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62161f4-3ed3-4fd6-9e83-5bb5c8dc757f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:27.127636 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62161f4-3ed3-4fd6-9e83-5bb5c8dc757f admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:27.127918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62161f4-3ed3-4fd6-9e83-5bb5c8dc757f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:27.128195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62161f4-3ed3-4fd6-9e83-5bb5c8dc757f admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:27.128902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62161f4-3ed3-4fd6-9e83-5bb5c8dc757f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:27.130783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62161f4-3ed3-4fd6-9e83-5bb5c8dc757f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:27.158945 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1277/2571] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:59:27 2022] POST /identity/v3/projects => generated 365 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:27.162843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7caab5d-d225-4f51-a4d0-8f99410de809 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:27.175923 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7caab5d-d225-4f51-a4d0-8f99410de809 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:27.231794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0014d586-7620-4492-a032-aa405da6c448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:27.232413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014d586-7620-4492-a032-aa405da6c448 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:27.232754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014d586-7620-4492-a032-aa405da6c448 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:27.233031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0014d586-7620-4492-a032-aa405da6c448 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:27.233700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0014d586-7620-4492-a032-aa405da6c448 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:27.235653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0014d586-7620-4492-a032-aa405da6c448 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:27.257172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1295/2572] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:59:27 2022] POST /identity/v3/users => generated 289 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:59:27.261519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62161f4-3ed3-4fd6-9e83-5bb5c8dc757f admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:27.271776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62161f4-3ed3-4fd6-9e83-5bb5c8dc757f admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:27.276837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf726c4-332b-47f6-9379-eabb50a07217 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:27.277450 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf726c4-332b-47f6-9379-eabb50a07217 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:27.277740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf726c4-332b-47f6-9379-eabb50a07217 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:27.278021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf726c4-332b-47f6-9379-eabb50a07217 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:27.278726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf726c4-332b-47f6-9379-eabb50a07217 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:27.280331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf726c4-332b-47f6-9379-eabb50a07217 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:27.291545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1278/2573] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:27.295755 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0014d586-7620-4492-a032-aa405da6c448 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:27.305576 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0014d586-7620-4492-a032-aa405da6c448 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:27.310960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55fcca3-4086-4dc4-824b-45589211f09a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:27.311566 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55fcca3-4086-4dc4-824b-45589211f09a admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:27.311857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55fcca3-4086-4dc4-824b-45589211f09a admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:27.312145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55fcca3-4086-4dc4-824b-45589211f09a admin admin] PATH_INFO: `/v3/projects/b7ba6df87dad46da8514ce6e898f5758/users/6d2523625e484b0ab4916cac259cf275/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:27.326805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55fcca3-4086-4dc4-824b-45589211f09a admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7ba6df87dad46da8514ce6e898f5758, user_id=6d2523625e484b0ab4916cac259cf275, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:27.329505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55fcca3-4086-4dc4-824b-45589211f09a admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:27.341809 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1296/2574] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:27 2022] PUT /identity/v3/projects/b7ba6df87dad46da8514ce6e898f5758/users/6d2523625e484b0ab4916cac259cf275/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:59:27.345787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf726c4-332b-47f6-9379-eabb50a07217 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:27.356551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf726c4-332b-47f6-9379-eabb50a07217 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:27.404004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:27.414830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1065fb08-02fe-456f-ab1b-ea1d624c4997 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:27.416025 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1065fb08-02fe-456f-ab1b-ea1d624c4997 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:27.416529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1065fb08-02fe-456f-ab1b-ea1d624c4997 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:27.416847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1065fb08-02fe-456f-ab1b-ea1d624c4997 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:27.417811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1065fb08-02fe-456f-ab1b-ea1d624c4997 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:27.419422 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1065fb08-02fe-456f-ab1b-ea1d624c4997 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:27.429367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1279/2575] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:27 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:27.435209 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55fcca3-4086-4dc4-824b-45589211f09a admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:27.450035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55fcca3-4086-4dc4-824b-45589211f09a admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:27.454706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3f98f3-a41b-46d0-a229-514caa01dfce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:27.455377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3f98f3-a41b-46d0-a229-514caa01dfce admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:27.455660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3f98f3-a41b-46d0-a229-514caa01dfce admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:27.455934 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3f98f3-a41b-46d0-a229-514caa01dfce admin admin] PATH_INFO: `/v3/projects/b7ba6df87dad46da8514ce6e898f5758/users/6d2523625e484b0ab4916cac259cf275/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:27.457911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3f98f3-a41b-46d0-a229-514caa01dfce admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7ba6df87dad46da8514ce6e898f5758, user_id=6d2523625e484b0ab4916cac259cf275, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:27.460527 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3f98f3-a41b-46d0-a229-514caa01dfce admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:27.471525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1297/2576] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:27 2022] PUT /identity/v3/projects/b7ba6df87dad46da8514ce6e898f5758/users/6d2523625e484b0ab4916cac259cf275/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:27.482217 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a0b7e5-b3e0-4dfc-a6fc-1fa352855179 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:27.482512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a0b7e5-b3e0-4dfc-a6fc-1fa352855179 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:27.482796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a0b7e5-b3e0-4dfc-a6fc-1fa352855179 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:27.510310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-a9a0b7e5-b3e0-4dfc-a6fc-1fa352855179 None None] MFA Rules not processed for user `6d2523625e484b0ab4916cac259cf275`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:27.511514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a9a0b7e5-b3e0-4dfc-a6fc-1fa352855179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:27.578661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1280/2577] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:27 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:27.590843 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3f98f3-a41b-46d0-a229-514caa01dfce admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:27.592544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7a3f98f3-a41b-46d0-a229-514caa01dfce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:27.614896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:27.671336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3f98f3-a41b-46d0-a229-514caa01dfce admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:27.675149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc06677-2b1d-4ff8-b24c-95ef6ea0cbf3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:27.675778 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc06677-2b1d-4ff8-b24c-95ef6ea0cbf3 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:27.676067 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc06677-2b1d-4ff8-b24c-95ef6ea0cbf3 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:27.676348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc06677-2b1d-4ff8-b24c-95ef6ea0cbf3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:27.686336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc06677-2b1d-4ff8-b24c-95ef6ea0cbf3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:27.687768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc06677-2b1d-4ff8-b24c-95ef6ea0cbf3 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:27.697749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1298/2578] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:27 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:27.771818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a0b7e5-b3e0-4dfc-a6fc-1fa352855179 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:27.782057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a0b7e5-b3e0-4dfc-a6fc-1fa352855179 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:27.785639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6553eb01-3b54-4cd9-998a-d7c9f09ad8fb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:27.786226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6553eb01-3b54-4cd9-998a-d7c9f09ad8fb service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:27.786509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6553eb01-3b54-4cd9-998a-d7c9f09ad8fb service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:27.786787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6553eb01-3b54-4cd9-998a-d7c9f09ad8fb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:27.844068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6553eb01-3b54-4cd9-998a-d7c9f09ad8fb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:27.845485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6553eb01-3b54-4cd9-998a-d7c9f09ad8fb service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:27.854909 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1281/2579] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:27 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:37.693021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=17 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:37.850620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=68 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:49.882492 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc06677-2b1d-4ff8-b24c-95ef6ea0cbf3 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:49.943710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc06677-2b1d-4ff8-b24c-95ef6ea0cbf3 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:49.948958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:49.949603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:49.949887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:49.950189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] PATH_INFO: `/v3/users/b2e058b9d30041e882040e8433b1f10d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:49.951358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2e058b9d30041e882040e8433b1f10d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:49.953300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:49.999494 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2e058b9d30041e882040e8433b1f10d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.000146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2e058b9d30041e882040e8433b1f10d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.000802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2e058b9d30041e882040e8433b1f10d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.013172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2e058b9d30041e882040e8433b1f10d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.020035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2e058b9d30041e882040e8433b1f10d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.032508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1299/2580] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:59:49 2022] DELETE /identity/v3/users/b2e058b9d30041e882040e8433b1f10d => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:59:50.037040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b5cbdf-d867-4e1d-bb52-e547307c6b4d None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:50.037504 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b5cbdf-d867-4e1d-bb52-e547307c6b4d None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:50.037646 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b5cbdf-d867-4e1d-bb52-e547307c6b4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:50.057373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-90b5cbdf-d867-4e1d-bb52-e547307c6b4d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:50.058791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-90b5cbdf-d867-4e1d-bb52-e547307c6b4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:50.117897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1282/2581] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:59:50.128140 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:50.129557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:50.186153 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56270fa4-57ce-4121-9d14-b182d2ef267c admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:50.190022 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586be565-be62-4ff3-ae17-15226a661834 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:50.190653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586be565-be62-4ff3-ae17-15226a661834 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:50.190940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586be565-be62-4ff3-ae17-15226a661834 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:50.191225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586be565-be62-4ff3-ae17-15226a661834 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:50.201236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586be565-be62-4ff3-ae17-15226a661834 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:50.202596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586be565-be62-4ff3-ae17-15226a661834 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:50.211907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1300/2582] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:50.480970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b5cbdf-d867-4e1d-bb52-e547307c6b4d None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:50.482541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-90b5cbdf-d867-4e1d-bb52-e547307c6b4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:50.495948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b5cbdf-d867-4e1d-bb52-e547307c6b4d None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:50.500649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:50.501309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:50.501599 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:50.501900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] PATH_INFO: `/v3/projects/e6aaf796c5db4f1ebbcb32e1b1c9010c` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:50.503059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e6aaf796c5db4f1ebbcb32e1b1c9010c)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:50.505048 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:50.552114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e6aaf796c5db4f1ebbcb32e1b1c9010c', 'request_id': 'req-a318cca3-88ef-44f2-92e4-3fe035fc999c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.560446 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e6aaf796c5db4f1ebbcb32e1b1c9010c', 'request_id': 'req-a318cca3-88ef-44f2-92e4-3fe035fc999c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.566055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e6aaf796c5db4f1ebbcb32e1b1c9010c', 'request_id': 'req-a318cca3-88ef-44f2-92e4-3fe035fc999c'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.577037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] The token cache is being invalidate because project e6aaf796c5db4f1ebbcb32e1b1c9010c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:59:50.577329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.577965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.579682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1283/2583] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:59:50 2022] DELETE /identity/v3/projects/e6aaf796c5db4f1ebbcb32e1b1c9010c => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:50.583774 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586be565-be62-4ff3-ae17-15226a661834 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:50.594136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586be565-be62-4ff3-ae17-15226a661834 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:50.613246 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:50.654133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:50.654942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:50.655257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:50.655991 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] PATH_INFO: `/v3/users/6d2523625e484b0ab4916cac259cf275` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:50.656821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d2523625e484b0ab4916cac259cf275)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:50.658968 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:50.705501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d2523625e484b0ab4916cac259cf275'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.706162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d2523625e484b0ab4916cac259cf275'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.706779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d2523625e484b0ab4916cac259cf275'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.720411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d2523625e484b0ab4916cac259cf275'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.727278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a5781f46-c092-4e4f-b034-b3fb3cd817fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d2523625e484b0ab4916cac259cf275'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:50.740941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1301/2584] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:59:50 2022] DELETE /identity/v3/users/6d2523625e484b0ab4916cac259cf275 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:51.009474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.011588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:51.027535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a318cca3-88ef-44f2-92e4-3fe035fc999c admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.085624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.086285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.086549 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.086865 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] PATH_INFO: `/v3/projects/b7ba6df87dad46da8514ce6e898f5758` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.088978 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7ba6df87dad46da8514ce6e898f5758)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:51.090622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:51.121681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:51.141417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7ba6df87dad46da8514ce6e898f5758', 'request_id': 'req-7e9865eb-a93a-49f6-a1f9-1995e460f66a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:51.149062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7ba6df87dad46da8514ce6e898f5758', 'request_id': 'req-7e9865eb-a93a-49f6-a1f9-1995e460f66a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:51.156994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7ba6df87dad46da8514ce6e898f5758', 'request_id': 'req-7e9865eb-a93a-49f6-a1f9-1995e460f66a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:51.168989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] The token cache is being invalidate because project b7ba6df87dad46da8514ce6e898f5758 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:59:51.168989 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:51.169662 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-7e9865eb-a93a-49f6-a1f9-1995e460f66a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:51.171197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1284/2585] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:59:51 2022] DELETE /identity/v3/projects/b7ba6df87dad46da8514ce6e898f5758 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:51.176059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f886dfa-cdad-4c6b-9ecb-3968b0355a8d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.176640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f886dfa-cdad-4c6b-9ecb-3968b0355a8d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.176640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f886dfa-cdad-4c6b-9ecb-3968b0355a8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.196884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-6f886dfa-cdad-4c6b-9ecb-3968b0355a8d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:51.198984 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-6f886dfa-cdad-4c6b-9ecb-3968b0355a8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:51.262061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1302/2586] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 04:59:51.269710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a09d518-5a1b-425a-acee-6a223086bf2c None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.270000 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a09d518-5a1b-425a-acee-6a223086bf2c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.270281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a09d518-5a1b-425a-acee-6a223086bf2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.291719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0a09d518-5a1b-425a-acee-6a223086bf2c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:51.293203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0a09d518-5a1b-425a-acee-6a223086bf2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:51.301382 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1285/2587] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:51.305596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f886dfa-cdad-4c6b-9ecb-3968b0355a8d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.315208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f886dfa-cdad-4c6b-9ecb-3968b0355a8d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.319695 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da696d1b-85a7-4ccd-ae43-5747d576bc4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.320296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da696d1b-85a7-4ccd-ae43-5747d576bc4a admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.320579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da696d1b-85a7-4ccd-ae43-5747d576bc4a admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.320882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da696d1b-85a7-4ccd-ae43-5747d576bc4a admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.321573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da696d1b-85a7-4ccd-ae43-5747d576bc4a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:59:51.321879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da696d1b-85a7-4ccd-ae43-5747d576bc4a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:51.324116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da696d1b-85a7-4ccd-ae43-5747d576bc4a admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:51.335316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1303/2588] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:59:51.340213 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a09d518-5a1b-425a-acee-6a223086bf2c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.349454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a09d518-5a1b-425a-acee-6a223086bf2c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.354254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e9ce50-5f3d-4f2b-8aff-9b3f2ee34320 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.354889 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e9ce50-5f3d-4f2b-8aff-9b3f2ee34320 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.355172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e9ce50-5f3d-4f2b-8aff-9b3f2ee34320 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.355449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e9ce50-5f3d-4f2b-8aff-9b3f2ee34320 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.356110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e9ce50-5f3d-4f2b-8aff-9b3f2ee34320 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:51.358027 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e9ce50-5f3d-4f2b-8aff-9b3f2ee34320 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:51.385728 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1286/2589] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:59:51 2022] POST /identity/v3/projects => generated 379 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:51.390051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da696d1b-85a7-4ccd-ae43-5747d576bc4a admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.400687 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da696d1b-85a7-4ccd-ae43-5747d576bc4a admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.426019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:51.458499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed12e819-89ad-4a84-8e98-a39eabb5e133 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.459117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed12e819-89ad-4a84-8e98-a39eabb5e133 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.459401 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed12e819-89ad-4a84-8e98-a39eabb5e133 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.459676 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed12e819-89ad-4a84-8e98-a39eabb5e133 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.460341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed12e819-89ad-4a84-8e98-a39eabb5e133 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:51.462272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed12e819-89ad-4a84-8e98-a39eabb5e133 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:51.482930 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1304/2590] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:59:51 2022] POST /identity/v3/users => generated 290 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:51.487041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e9ce50-5f3d-4f2b-8aff-9b3f2ee34320 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.498269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e9ce50-5f3d-4f2b-8aff-9b3f2ee34320 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.503065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e2d04b-31f3-4481-8e29-a28d7b74f303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.503681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e2d04b-31f3-4481-8e29-a28d7b74f303 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.503968 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e2d04b-31f3-4481-8e29-a28d7b74f303 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.504246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e2d04b-31f3-4481-8e29-a28d7b74f303 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.504987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e2d04b-31f3-4481-8e29-a28d7b74f303 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:51.506592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e2d04b-31f3-4481-8e29-a28d7b74f303 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:51.517381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1287/2591] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:59:51.521142 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed12e819-89ad-4a84-8e98-a39eabb5e133 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.531018 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed12e819-89ad-4a84-8e98-a39eabb5e133 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.536546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09232f71-0039-47ab-a3a9-d94e9a98f34d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.537185 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09232f71-0039-47ab-a3a9-d94e9a98f34d admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.537470 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09232f71-0039-47ab-a3a9-d94e9a98f34d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.537747 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09232f71-0039-47ab-a3a9-d94e9a98f34d admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.538439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09232f71-0039-47ab-a3a9-d94e9a98f34d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:51.540623 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09232f71-0039-47ab-a3a9-d94e9a98f34d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:51.552197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1305/2592] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:59:51.556320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e2d04b-31f3-4481-8e29-a28d7b74f303 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.567823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e2d04b-31f3-4481-8e29-a28d7b74f303 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.573557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d18ff8-9d56-4b0d-bf7d-e36c33992ef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.574229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d18ff8-9d56-4b0d-bf7d-e36c33992ef9 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.574518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d18ff8-9d56-4b0d-bf7d-e36c33992ef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.574794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d18ff8-9d56-4b0d-bf7d-e36c33992ef9 admin admin] PATH_INFO: `/v3/projects/387f625d07b746cd9a5b9d6aaaf4521d/users/91869e02194848fe90c4d809108ffaea/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.590325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d18ff8-9d56-4b0d-bf7d-e36c33992ef9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=387f625d07b746cd9a5b9d6aaaf4521d, user_id=91869e02194848fe90c4d809108ffaea, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:51.593067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d18ff8-9d56-4b0d-bf7d-e36c33992ef9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:51.606006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1288/2593] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:51 2022] PUT /identity/v3/projects/387f625d07b746cd9a5b9d6aaaf4521d/users/91869e02194848fe90c4d809108ffaea/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:51.617345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b26be-3432-410d-9af7-992847b85b17 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.617637 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b26be-3432-410d-9af7-992847b85b17 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.617920 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36b26be-3432-410d-9af7-992847b85b17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.646777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c36b26be-3432-410d-9af7-992847b85b17 None None] MFA Rules not processed for user `91869e02194848fe90c4d809108ffaea`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:51.648031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c36b26be-3432-410d-9af7-992847b85b17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:51.715149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1306/2594] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:51 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:51.719982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d18ff8-9d56-4b0d-bf7d-e36c33992ef9 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.783648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d18ff8-9d56-4b0d-bf7d-e36c33992ef9 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.788794 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2510508f-5887-4fd7-9618-decd5ec6d31e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.789570 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2510508f-5887-4fd7-9618-decd5ec6d31e admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.789859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2510508f-5887-4fd7-9618-decd5ec6d31e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.790138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2510508f-5887-4fd7-9618-decd5ec6d31e admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.791292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2510508f-5887-4fd7-9618-decd5ec6d31e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:51.794419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2510508f-5887-4fd7-9618-decd5ec6d31e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:51.823138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1289/2595] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:59:51 2022] POST /identity/v3/projects => generated 381 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:51.827232 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b26be-3432-410d-9af7-992847b85b17 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.838015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36b26be-3432-410d-9af7-992847b85b17 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.902699 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce283207-864d-420d-9acb-d996055c5113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.903452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce283207-864d-420d-9acb-d996055c5113 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.903781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce283207-864d-420d-9acb-d996055c5113 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.904124 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce283207-864d-420d-9acb-d996055c5113 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.904854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce283207-864d-420d-9acb-d996055c5113 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:51.906860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce283207-864d-420d-9acb-d996055c5113 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:51.931363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1307/2596] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:59:51 2022] POST /identity/v3/users => generated 297 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:51.936026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2510508f-5887-4fd7-9618-decd5ec6d31e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.947630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2510508f-5887-4fd7-9618-decd5ec6d31e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.953684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50579856-59fd-450f-b090-2f044224dc97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.954499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50579856-59fd-450f-b090-2f044224dc97 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.954900 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50579856-59fd-450f-b090-2f044224dc97 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.955434 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50579856-59fd-450f-b090-2f044224dc97 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:51.956300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50579856-59fd-450f-b090-2f044224dc97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:51.959002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50579856-59fd-450f-b090-2f044224dc97 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:51.972665 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1290/2597] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:51 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:59:51.976493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce283207-864d-420d-9acb-d996055c5113 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:51.988469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce283207-864d-420d-9acb-d996055c5113 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:51.993610 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a68922-8737-4800-a02c-df9f3c4368cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:51.995046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a68922-8737-4800-a02c-df9f3c4368cc admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:51.995376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a68922-8737-4800-a02c-df9f3c4368cc admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:51.995849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a68922-8737-4800-a02c-df9f3c4368cc admin admin] PATH_INFO: `/v3/projects/83564e4113f14df9a99b70e352b6735c/users/5861bab9b7444776805854e2afc1374f/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:52.013722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a68922-8737-4800-a02c-df9f3c4368cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=83564e4113f14df9a99b70e352b6735c, user_id=5861bab9b7444776805854e2afc1374f, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:52.016485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a68922-8737-4800-a02c-df9f3c4368cc admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:52.029432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1308/2598] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:51 2022] PUT /identity/v3/projects/83564e4113f14df9a99b70e352b6735c/users/5861bab9b7444776805854e2afc1374f/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:52.033500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50579856-59fd-450f-b090-2f044224dc97 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:52.044544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50579856-59fd-450f-b090-2f044224dc97 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:52.064490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:52.102339 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0ca352-8e50-4b73-8e06-62276f3ce1b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:52.102966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0ca352-8e50-4b73-8e06-62276f3ce1b9 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:52.103233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0ca352-8e50-4b73-8e06-62276f3ce1b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:52.104055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0ca352-8e50-4b73-8e06-62276f3ce1b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:52.104875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0ca352-8e50-4b73-8e06-62276f3ce1b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:52.106498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0ca352-8e50-4b73-8e06-62276f3ce1b9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:52.116151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1291/2599] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:52 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:52.120588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a68922-8737-4800-a02c-df9f3c4368cc admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:52.126064 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:52.132418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a68922-8737-4800-a02c-df9f3c4368cc admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:52.138081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429b1eee-cab4-4a9a-ba5d-125761d69827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:52.138781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429b1eee-cab4-4a9a-ba5d-125761d69827 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:52.139074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429b1eee-cab4-4a9a-ba5d-125761d69827 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:52.139362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429b1eee-cab4-4a9a-ba5d-125761d69827 admin admin] PATH_INFO: `/v3/projects/83564e4113f14df9a99b70e352b6735c/users/5861bab9b7444776805854e2afc1374f/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:52.141436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429b1eee-cab4-4a9a-ba5d-125761d69827 admin admin] RBAC: Authorizing `identity:create_grant(project_id=83564e4113f14df9a99b70e352b6735c, user_id=5861bab9b7444776805854e2afc1374f, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:52.144276 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429b1eee-cab4-4a9a-ba5d-125761d69827 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:52.155712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1309/2600] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:52 2022] PUT /identity/v3/projects/83564e4113f14df9a99b70e352b6735c/users/5861bab9b7444776805854e2afc1374f/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:52.166777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1087b7c8-76ba-40ae-a023-484d6d8d1373 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:52.167100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1087b7c8-76ba-40ae-a023-484d6d8d1373 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:52.167345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1087b7c8-76ba-40ae-a023-484d6d8d1373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:52.197148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-1087b7c8-76ba-40ae-a023-484d6d8d1373 None None] MFA Rules not processed for user `5861bab9b7444776805854e2afc1374f`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:52.198378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-1087b7c8-76ba-40ae-a023-484d6d8d1373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:52.278388 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1292/2601] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:52 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:52.289815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429b1eee-cab4-4a9a-ba5d-125761d69827 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:52.293643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-429b1eee-cab4-4a9a-ba5d-125761d69827 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:52.351632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429b1eee-cab4-4a9a-ba5d-125761d69827 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:52.355924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc33b6e-1dfb-4933-b3f9-df9ba133fff2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:52.356557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc33b6e-1dfb-4933-b3f9-df9ba133fff2 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:52.356878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc33b6e-1dfb-4933-b3f9-df9ba133fff2 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:52.357156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc33b6e-1dfb-4933-b3f9-df9ba133fff2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:52.366331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc33b6e-1dfb-4933-b3f9-df9ba133fff2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:52.367776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc33b6e-1dfb-4933-b3f9-df9ba133fff2 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:52.378202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1310/2602] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:52 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:52.776580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1087b7c8-76ba-40ae-a023-484d6d8d1373 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:52.839347 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1087b7c8-76ba-40ae-a023-484d6d8d1373 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:52.844243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:52.844921 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:52.845213 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:52.845527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] PATH_INFO: `/v3/users/91869e02194848fe90c4d809108ffaea` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:52.846704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] RBAC: Authorizing `identity:delete_user(user_id=91869e02194848fe90c4d809108ffaea)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:52.848697 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:52.896212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91869e02194848fe90c4d809108ffaea'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:52.897879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:52.904679 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91869e02194848fe90c4d809108ffaea'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:52.905598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91869e02194848fe90c4d809108ffaea'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:52.906332 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91869e02194848fe90c4d809108ffaea'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:52.913386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91869e02194848fe90c4d809108ffaea'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:52.930802 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1293/2603] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:59:52 2022] DELETE /identity/v3/users/91869e02194848fe90c4d809108ffaea => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:52.935442 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199238a4-bd9d-41a4-a96c-88a2514d2f9d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:52.935736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199238a4-bd9d-41a4-a96c-88a2514d2f9d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:52.936019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199238a4-bd9d-41a4-a96c-88a2514d2f9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:52.955775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-199238a4-bd9d-41a4-a96c-88a2514d2f9d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:52.957012 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-199238a4-bd9d-41a4-a96c-88a2514d2f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:53.015173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1311/2604] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:53.024844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:53.026391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:53.088970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442da964-ea3f-4a14-a8db-7394619e9670 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:53.092833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e25b2f0-eac7-4559-bdd8-69aa4809dddb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:53.093394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e25b2f0-eac7-4559-bdd8-69aa4809dddb service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:53.093684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e25b2f0-eac7-4559-bdd8-69aa4809dddb service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:53.093965 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e25b2f0-eac7-4559-bdd8-69aa4809dddb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:53.103533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e25b2f0-eac7-4559-bdd8-69aa4809dddb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:53.105030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e25b2f0-eac7-4559-bdd8-69aa4809dddb service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:53.115035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1294/2605] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:53.401804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199238a4-bd9d-41a4-a96c-88a2514d2f9d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:53.403404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-199238a4-bd9d-41a4-a96c-88a2514d2f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:53.416419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199238a4-bd9d-41a4-a96c-88a2514d2f9d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:53.421542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:53.422203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:53.422485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:53.422761 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] PATH_INFO: `/v3/projects/387f625d07b746cd9a5b9d6aaaf4521d` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:53.423951 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] RBAC: Authorizing `identity:delete_project(project_id=387f625d07b746cd9a5b9d6aaaf4521d)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:53.425948 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:53.473154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '387f625d07b746cd9a5b9d6aaaf4521d', 'request_id': 'req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:53.479369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '387f625d07b746cd9a5b9d6aaaf4521d', 'request_id': 'req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:53.487403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '387f625d07b746cd9a5b9d6aaaf4521d', 'request_id': 'req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:53.498449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] The token cache is being invalidate because project 387f625d07b746cd9a5b9d6aaaf4521d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:59:53.498738 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:53.499362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:53.500894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1312/2606] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:59:53 2022] DELETE /identity/v3/projects/387f625d07b746cd9a5b9d6aaaf4521d => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:53.504981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e25b2f0-eac7-4559-bdd8-69aa4809dddb service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:53.515320 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e25b2f0-eac7-4559-bdd8-69aa4809dddb service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:53.581481 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:53.582152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:53.582546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:53.582837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] PATH_INFO: `/v3/users/5861bab9b7444776805854e2afc1374f` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:53.584033 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5861bab9b7444776805854e2afc1374f)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:53.585998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:53.636714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5861bab9b7444776805854e2afc1374f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:53.646932 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5861bab9b7444776805854e2afc1374f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:53.649258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5861bab9b7444776805854e2afc1374f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:53.650184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5861bab9b7444776805854e2afc1374f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:53.660434 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-bca8081c-a9cf-4c2e-8492-82a8017978f7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5861bab9b7444776805854e2afc1374f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:53.666488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:INSERT=2,keystone:DELETE=12,keystone:SELECT=86 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:53.683837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1295/2607] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:59:53 2022] DELETE /identity/v3/users/5861bab9b7444776805854e2afc1374f => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:54.019069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:54.020574 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:54.034535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd015fa3-e6d9-4a0d-8b40-b7e4bd8b5acd admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:54.047727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:54.097044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:54.097697 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.097980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.098260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] PATH_INFO: `/v3/projects/83564e4113f14df9a99b70e352b6735c` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.099451 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] RBAC: Authorizing `identity:delete_project(project_id=83564e4113f14df9a99b70e352b6735c)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:54.101426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:54.148586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '83564e4113f14df9a99b70e352b6735c', 'request_id': 'req-2b25d999-ddc1-4127-97d3-f6418e48996b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:54.155433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '83564e4113f14df9a99b70e352b6735c', 'request_id': 'req-2b25d999-ddc1-4127-97d3-f6418e48996b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:54.162760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '83564e4113f14df9a99b70e352b6735c', 'request_id': 'req-2b25d999-ddc1-4127-97d3-f6418e48996b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:54.174436 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] The token cache is being invalidate because project 83564e4113f14df9a99b70e352b6735c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:59:54.174735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:54.175382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-2b25d999-ddc1-4127-97d3-f6418e48996b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:54.176937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1313/2608] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:59:54 2022] DELETE /identity/v3/projects/83564e4113f14df9a99b70e352b6735c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:54.182299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8877f9c-1bd1-46ef-8967-7d49be67f49c None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.182781 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8877f9c-1bd1-46ef-8967-7d49be67f49c None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.182877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8877f9c-1bd1-46ef-8967-7d49be67f49c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.202659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-b8877f9c-1bd1-46ef-8967-7d49be67f49c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:54.203886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b8877f9c-1bd1-46ef-8967-7d49be67f49c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:54.262299 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1296/2609] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:54.271250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c9857b-bc6b-44d6-93fe-a6d9a5a057c4 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.271553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c9857b-bc6b-44d6-93fe-a6d9a5a057c4 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.271840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c9857b-bc6b-44d6-93fe-a6d9a5a057c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.293775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-e1c9857b-bc6b-44d6-93fe-a6d9a5a057c4 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:54.295152 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e1c9857b-bc6b-44d6-93fe-a6d9a5a057c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:54.303776 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1314/2610] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:54.308453 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8877f9c-1bd1-46ef-8967-7d49be67f49c None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:54.319698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8877f9c-1bd1-46ef-8967-7d49be67f49c None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:54.324957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad55f687-c719-4d31-b44c-731e9b061e67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:54.325761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad55f687-c719-4d31-b44c-731e9b061e67 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.326020 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad55f687-c719-4d31-b44c-731e9b061e67 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.326819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad55f687-c719-4d31-b44c-731e9b061e67 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.327156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad55f687-c719-4d31-b44c-731e9b061e67 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:59:54.327541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad55f687-c719-4d31-b44c-731e9b061e67 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:54.329254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad55f687-c719-4d31-b44c-731e9b061e67 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:54.344689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1297/2611] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:59:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:59:54.347341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c9857b-bc6b-44d6-93fe-a6d9a5a057c4 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:54.357951 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c9857b-bc6b-44d6-93fe-a6d9a5a057c4 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:54.362815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0039698-24eb-4e42-8775-adfa69566e6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:54.363439 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0039698-24eb-4e42-8775-adfa69566e6f admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.363725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0039698-24eb-4e42-8775-adfa69566e6f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.364004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0039698-24eb-4e42-8775-adfa69566e6f admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.364838 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0039698-24eb-4e42-8775-adfa69566e6f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:54.366722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0039698-24eb-4e42-8775-adfa69566e6f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:54.421031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1315/2612] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:59:54 2022] POST /identity/v3/projects => generated 373 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:54.424997 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad55f687-c719-4d31-b44c-731e9b061e67 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:54.437004 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad55f687-c719-4d31-b44c-731e9b061e67 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:54.507982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68109cc6-ac16-4709-b220-2aa038213ad9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:54.508936 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68109cc6-ac16-4709-b220-2aa038213ad9 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.509393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68109cc6-ac16-4709-b220-2aa038213ad9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.510474 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68109cc6-ac16-4709-b220-2aa038213ad9 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.511324 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68109cc6-ac16-4709-b220-2aa038213ad9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:54.513445 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68109cc6-ac16-4709-b220-2aa038213ad9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:54.535598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1298/2613] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:59:54 2022] POST /identity/v3/users => generated 287 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:54.539731 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0039698-24eb-4e42-8775-adfa69566e6f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:54.550691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0039698-24eb-4e42-8775-adfa69566e6f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:54.555307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b7acc-937c-4a3f-bc3c-011420aa0f65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:54.555914 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b7acc-937c-4a3f-bc3c-011420aa0f65 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.556203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b7acc-937c-4a3f-bc3c-011420aa0f65 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.556667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6b7acc-937c-4a3f-bc3c-011420aa0f65 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.557413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b7acc-937c-4a3f-bc3c-011420aa0f65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:54.559007 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6b7acc-937c-4a3f-bc3c-011420aa0f65 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:54.569476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1316/2614] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:59:54.574304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68109cc6-ac16-4709-b220-2aa038213ad9 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:54.584708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68109cc6-ac16-4709-b220-2aa038213ad9 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:54.590605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b745a2-f269-4016-badd-14d78f1d25a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:54.591250 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b745a2-f269-4016-badd-14d78f1d25a6 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.591534 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b745a2-f269-4016-badd-14d78f1d25a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.591828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b745a2-f269-4016-badd-14d78f1d25a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.592537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b745a2-f269-4016-badd-14d78f1d25a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:54.594201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b745a2-f269-4016-badd-14d78f1d25a6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:54.605306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1299/2615] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:54.609664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b7acc-937c-4a3f-bc3c-011420aa0f65 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:54.619535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6b7acc-937c-4a3f-bc3c-011420aa0f65 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:54.624121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfe93bd-4ed4-49b4-9678-f38385aca049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:54.624804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfe93bd-4ed4-49b4-9678-f38385aca049 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.625088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfe93bd-4ed4-49b4-9678-f38385aca049 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.625366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfe93bd-4ed4-49b4-9678-f38385aca049 admin admin] PATH_INFO: `/v3/projects/593bfcdbff2d4cbeb4d326c970d3606c/users/fdebc3ec0ae34cd5b4d80f920327e311/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.640255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfe93bd-4ed4-49b4-9678-f38385aca049 admin admin] RBAC: Authorizing `identity:create_grant(project_id=593bfcdbff2d4cbeb4d326c970d3606c, user_id=fdebc3ec0ae34cd5b4d80f920327e311, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:54.642908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfe93bd-4ed4-49b4-9678-f38385aca049 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:54.658640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1317/2616] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:54 2022] PUT /identity/v3/projects/593bfcdbff2d4cbeb4d326c970d3606c/users/fdebc3ec0ae34cd5b4d80f920327e311/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:54.669086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe9c626-fff1-412a-bd34-4878a5444f5e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.669316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe9c626-fff1-412a-bd34-4878a5444f5e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.669598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe9c626-fff1-412a-bd34-4878a5444f5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.697937 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4fe9c626-fff1-412a-bd34-4878a5444f5e None None] MFA Rules not processed for user `fdebc3ec0ae34cd5b4d80f920327e311`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:54.699499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4fe9c626-fff1-412a-bd34-4878a5444f5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:54.739947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:54.764879 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1300/2617] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:54 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:54.777386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfe93bd-4ed4-49b4-9678-f38385aca049 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:54.779089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7bfe93bd-4ed4-49b4-9678-f38385aca049 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:54.826282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:54.839148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfe93bd-4ed4-49b4-9678-f38385aca049 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:54.842829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c89ec8c-85fe-4aa8-b8b1-63caeebc8843 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:54.843480 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c89ec8c-85fe-4aa8-b8b1-63caeebc8843 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.843768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c89ec8c-85fe-4aa8-b8b1-63caeebc8843 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.844071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c89ec8c-85fe-4aa8-b8b1-63caeebc8843 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.852299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c89ec8c-85fe-4aa8-b8b1-63caeebc8843 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:54.853802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c89ec8c-85fe-4aa8-b8b1-63caeebc8843 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:54.862518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1318/2618] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:54 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:54.884457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe9c626-fff1-412a-bd34-4878a5444f5e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:54.947051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe9c626-fff1-412a-bd34-4878a5444f5e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:54.954060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:54.954712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:54.954999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:54.955302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] PATH_INFO: `/v3/users/fdebc3ec0ae34cd5b4d80f920327e311` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:54.956498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fdebc3ec0ae34cd5b4d80f920327e311)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:54.958457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:55.004527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fdebc3ec0ae34cd5b4d80f920327e311'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:55.012233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fdebc3ec0ae34cd5b4d80f920327e311'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:55.013035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fdebc3ec0ae34cd5b4d80f920327e311'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:55.013696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fdebc3ec0ae34cd5b4d80f920327e311'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:55.020925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fdebc3ec0ae34cd5b4d80f920327e311'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:55.038698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1301/2619] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:59:54 2022] DELETE /identity/v3/users/fdebc3ec0ae34cd5b4d80f920327e311 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:55.043575 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2f92e-f4a6-41d1-a164-0453a3166ec5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.043919 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2f92e-f4a6-41d1-a164-0453a3166ec5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.044205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f2f92e-f4a6-41d1-a164-0453a3166ec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:55.063924 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-75f2f92e-f4a6-41d1-a164-0453a3166ec5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:55.065190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-75f2f92e-f4a6-41d1-a164-0453a3166ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:55.122967 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1319/2620] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:55.132380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:55.133831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:55.193244 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce870a6-39d0-4fad-b2ac-444ad88df989 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:55.196837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca16dd8-824c-44ed-b4a5-2350288b7b83 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:55.197661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca16dd8-824c-44ed-b4a5-2350288b7b83 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.197946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca16dd8-824c-44ed-b4a5-2350288b7b83 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.198228 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca16dd8-824c-44ed-b4a5-2350288b7b83 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:55.207317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca16dd8-824c-44ed-b4a5-2350288b7b83 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:55.208789 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca16dd8-824c-44ed-b4a5-2350288b7b83 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:55.218389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1302/2621] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:55.493741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2f92e-f4a6-41d1-a164-0453a3166ec5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:55.496822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-75f2f92e-f4a6-41d1-a164-0453a3166ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:55.510216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f2f92e-f4a6-41d1-a164-0453a3166ec5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:55.514940 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:55.515840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.516205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.516561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] PATH_INFO: `/v3/projects/593bfcdbff2d4cbeb4d326c970d3606c` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:55.518003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] RBAC: Authorizing `identity:delete_project(project_id=593bfcdbff2d4cbeb4d326c970d3606c)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:55.520399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:55.569324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '593bfcdbff2d4cbeb4d326c970d3606c', 'request_id': 'req-c49a0793-58de-49e7-8012-df240327382f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:55.575661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '593bfcdbff2d4cbeb4d326c970d3606c', 'request_id': 'req-c49a0793-58de-49e7-8012-df240327382f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:55.583090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '593bfcdbff2d4cbeb4d326c970d3606c', 'request_id': 'req-c49a0793-58de-49e7-8012-df240327382f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:55.594725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] The token cache is being invalidate because project 593bfcdbff2d4cbeb4d326c970d3606c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:59:55.595019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:55.595647 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-c49a0793-58de-49e7-8012-df240327382f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:55.597200 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1320/2622] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:59:55 2022] DELETE /identity/v3/projects/593bfcdbff2d4cbeb4d326c970d3606c => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:55.602126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87692e-f6fb-4186-886b-3c6c14f6f890 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.602471 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87692e-f6fb-4186-886b-3c6c14f6f890 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.602770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87692e-f6fb-4186-886b-3c6c14f6f890 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:55.612088 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:55.623574 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-bd87692e-f6fb-4186-886b-3c6c14f6f890 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:55.625018 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-bd87692e-f6fb-4186-886b-3c6c14f6f890 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:55.685822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1303/2623] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:55.693677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59708c53-0b7e-4152-9aab-854b6b3ea8d5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.693976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59708c53-0b7e-4152-9aab-854b6b3ea8d5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.694265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59708c53-0b7e-4152-9aab-854b6b3ea8d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:55.714709 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-59708c53-0b7e-4152-9aab-854b6b3ea8d5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:55.715857 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-59708c53-0b7e-4152-9aab-854b6b3ea8d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:55.723828 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1321/2624] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:55.727946 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd87692e-f6fb-4186-886b-3c6c14f6f890 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:55.738174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd87692e-f6fb-4186-886b-3c6c14f6f890 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:55.742785 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cc4a2f-85a7-4740-b9d0-1cb6992512d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:55.743381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cc4a2f-85a7-4740-b9d0-1cb6992512d1 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.743669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cc4a2f-85a7-4740-b9d0-1cb6992512d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.743977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cc4a2f-85a7-4740-b9d0-1cb6992512d1 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:55.745007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cc4a2f-85a7-4740-b9d0-1cb6992512d1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:59:55.745319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cc4a2f-85a7-4740-b9d0-1cb6992512d1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:55.746911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cc4a2f-85a7-4740-b9d0-1cb6992512d1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:55.758731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1304/2625] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:59:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:59:55.762798 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59708c53-0b7e-4152-9aab-854b6b3ea8d5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:55.772036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59708c53-0b7e-4152-9aab-854b6b3ea8d5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:55.777448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f940f58-f555-4fd7-8743-fd5aea31c443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:55.778059 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f940f58-f555-4fd7-8743-fd5aea31c443 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.778345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f940f58-f555-4fd7-8743-fd5aea31c443 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.778645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f940f58-f555-4fd7-8743-fd5aea31c443 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:55.779311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f940f58-f555-4fd7-8743-fd5aea31c443 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:55.781284 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f940f58-f555-4fd7-8743-fd5aea31c443 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:55.809867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1322/2626] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:59:55 2022] POST /identity/v3/projects => generated 361 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 04:59:55.814039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cc4a2f-85a7-4740-b9d0-1cb6992512d1 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:55.824452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cc4a2f-85a7-4740-b9d0-1cb6992512d1 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:55.881054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ee1d6-3bce-4be2-b7dc-9583ad30f2a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:55.881676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ee1d6-3bce-4be2-b7dc-9583ad30f2a5 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.881990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ee1d6-3bce-4be2-b7dc-9583ad30f2a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.882272 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ee1d6-3bce-4be2-b7dc-9583ad30f2a5 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:55.882947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3ee1d6-3bce-4be2-b7dc-9583ad30f2a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:55.885472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3ee1d6-3bce-4be2-b7dc-9583ad30f2a5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:55.906759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1305/2627] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:59:55 2022] POST /identity/v3/users => generated 281 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:55.910915 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f940f58-f555-4fd7-8743-fd5aea31c443 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:55.921717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f940f58-f555-4fd7-8743-fd5aea31c443 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:55.926393 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c2b9bb-45b5-4c4d-859e-a388752551f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:55.927015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c2b9bb-45b5-4c4d-859e-a388752551f3 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.927308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c2b9bb-45b5-4c4d-859e-a388752551f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.927594 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c2b9bb-45b5-4c4d-859e-a388752551f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:55.928292 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c2b9bb-45b5-4c4d-859e-a388752551f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:55.929928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c2b9bb-45b5-4c4d-859e-a388752551f3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:55.940128 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1323/2628] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:55.943816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ee1d6-3bce-4be2-b7dc-9583ad30f2a5 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:55.954804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ee1d6-3bce-4be2-b7dc-9583ad30f2a5 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:55.959447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3f498d-a2db-40a5-b2ca-7a5880903d1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:55.960052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3f498d-a2db-40a5-b2ca-7a5880903d1d admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.960336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3f498d-a2db-40a5-b2ca-7a5880903d1d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.960640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3f498d-a2db-40a5-b2ca-7a5880903d1d admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:55.961338 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3f498d-a2db-40a5-b2ca-7a5880903d1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:55.963567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3f498d-a2db-40a5-b2ca-7a5880903d1d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:55.974361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1306/2629] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:55.978515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c2b9bb-45b5-4c4d-859e-a388752551f3 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:55.988319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c2b9bb-45b5-4c4d-859e-a388752551f3 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:55.992936 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c78ce8-befd-46d4-bfe5-0fb3a4c68220 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:55.993600 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c78ce8-befd-46d4-bfe5-0fb3a4c68220 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:55.993943 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c78ce8-befd-46d4-bfe5-0fb3a4c68220 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:55.994263 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c78ce8-befd-46d4-bfe5-0fb3a4c68220 admin admin] PATH_INFO: `/v3/projects/9c349a66bb034f4783871d9c3289fe01/users/25326b7da9a14052969be38b3d07ee21/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:56.009392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c78ce8-befd-46d4-bfe5-0fb3a4c68220 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c349a66bb034f4783871d9c3289fe01, user_id=25326b7da9a14052969be38b3d07ee21, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:56.012878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c78ce8-befd-46d4-bfe5-0fb3a4c68220 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:56.026686 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1324/2630] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:55 2022] PUT /identity/v3/projects/9c349a66bb034f4783871d9c3289fe01/users/25326b7da9a14052969be38b3d07ee21/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:59:56.041408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65280f4d-56d7-4d32-9b81-46d634ec5106 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:56.041699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65280f4d-56d7-4d32-9b81-46d634ec5106 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:56.041974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65280f4d-56d7-4d32-9b81-46d634ec5106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:56.069363 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-65280f4d-56d7-4d32-9b81-46d634ec5106 None None] MFA Rules not processed for user `25326b7da9a14052969be38b3d07ee21`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:56.070590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-65280f4d-56d7-4d32-9b81-46d634ec5106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:56.130936 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:56.136156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1307/2631] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:56 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:56.168810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c78ce8-befd-46d4-bfe5-0fb3a4c68220 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:56.170537 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-71c78ce8-befd-46d4-bfe5-0fb3a4c68220 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:56.177068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:56.229779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c78ce8-befd-46d4-bfe5-0fb3a4c68220 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:56.233410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70b8e31-155d-4592-9877-3fd576d2481f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:56.234042 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70b8e31-155d-4592-9877-3fd576d2481f service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:56.234759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70b8e31-155d-4592-9877-3fd576d2481f service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:56.234759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70b8e31-155d-4592-9877-3fd576d2481f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:56.243963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70b8e31-155d-4592-9877-3fd576d2481f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:56.245446 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70b8e31-155d-4592-9877-3fd576d2481f service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:56.254265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1325/2632] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:56 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:56.269456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65280f4d-56d7-4d32-9b81-46d634ec5106 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:56.331993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65280f4d-56d7-4d32-9b81-46d634ec5106 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:56.336462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:56.337129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:56.337413 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:56.337688 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] PATH_INFO: `/v3/users/25326b7da9a14052969be38b3d07ee21` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:56.338850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25326b7da9a14052969be38b3d07ee21)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:56.340791 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:56.385654 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25326b7da9a14052969be38b3d07ee21'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:56.393212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25326b7da9a14052969be38b3d07ee21'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:56.393853 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25326b7da9a14052969be38b3d07ee21'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:56.394469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25326b7da9a14052969be38b3d07ee21'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:56.403146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25326b7da9a14052969be38b3d07ee21'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:56.421008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1308/2633] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 04:59:56 2022] DELETE /identity/v3/users/25326b7da9a14052969be38b3d07ee21 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:56.425625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7e1812-f330-44da-bceb-fb62c6ca8394 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:56.425926 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7e1812-f330-44da-bceb-fb62c6ca8394 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:56.426213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7e1812-f330-44da-bceb-fb62c6ca8394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:56.445571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-3a7e1812-f330-44da-bceb-fb62c6ca8394 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:56.446802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3a7e1812-f330-44da-bceb-fb62c6ca8394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:56.505660 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1326/2634] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:56.516002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:56.517548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:56.576922 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec98e67-d804-45e5-a4ec-8958de8a79d7 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:56.580641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4242065-66f7-4e0f-b390-279f58fce1e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:56.581267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4242065-66f7-4e0f-b390-279f58fce1e6 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:56.581557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4242065-66f7-4e0f-b390-279f58fce1e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:56.581838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4242065-66f7-4e0f-b390-279f58fce1e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:56.591281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4242065-66f7-4e0f-b390-279f58fce1e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:56.592744 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4242065-66f7-4e0f-b390-279f58fce1e6 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:56.602485 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1309/2635] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 04:59:56.866348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7e1812-f330-44da-bceb-fb62c6ca8394 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:56.867835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3a7e1812-f330-44da-bceb-fb62c6ca8394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:56.880586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7e1812-f330-44da-bceb-fb62c6ca8394 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:56.887983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:56.889072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:56.889360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:56.889641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] PATH_INFO: `/v3/projects/9c349a66bb034f4783871d9c3289fe01` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:56.890823 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c349a66bb034f4783871d9c3289fe01)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:56.892794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:56.940291 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c349a66bb034f4783871d9c3289fe01', 'request_id': 'req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:56.946462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c349a66bb034f4783871d9c3289fe01', 'request_id': 'req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:56.954679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c349a66bb034f4783871d9c3289fe01', 'request_id': 'req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:56.965499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] The token cache is being invalidate because project 9c349a66bb034f4783871d9c3289fe01 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 04:59:56.965795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:56.966475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6d0b0043-f026-40b9-acc0-2d0ad576bbf8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 04:59:56.967986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1327/2636] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 04:59:56 2022] DELETE /identity/v3/projects/9c349a66bb034f4783871d9c3289fe01 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:56.972897 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bf585a-da3f-4dd6-abfb-1e2c62a6d18a None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:56.973195 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bf585a-da3f-4dd6-abfb-1e2c62a6d18a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:56.973506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bf585a-da3f-4dd6-abfb-1e2c62a6d18a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:56.993518 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-77bf585a-da3f-4dd6-abfb-1e2c62a6d18a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:56.994761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-77bf585a-da3f-4dd6-abfb-1e2c62a6d18a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:57.012143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:57.056223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1310/2637] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:57.064409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dac204-91f4-4435-9747-ea5a6a72a070 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.064986 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dac204-91f4-4435-9747-ea5a6a72a070 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.065354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dac204-91f4-4435-9747-ea5a6a72a070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.085872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f6dac204-91f4-4435-9747-ea5a6a72a070 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:57.087083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f6dac204-91f4-4435-9747-ea5a6a72a070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:57.095192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1328/2638] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:57.099387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bf585a-da3f-4dd6-abfb-1e2c62a6d18a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.109284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bf585a-da3f-4dd6-abfb-1e2c62a6d18a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.113930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84836b9e-6c89-43f6-8d02-817c5205c705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.114544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84836b9e-6c89-43f6-8d02-817c5205c705 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.114845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84836b9e-6c89-43f6-8d02-817c5205c705 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.115125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84836b9e-6c89-43f6-8d02-817c5205c705 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.115858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84836b9e-6c89-43f6-8d02-817c5205c705 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 04:59:57.116173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84836b9e-6c89-43f6-8d02-817c5205c705 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.117822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84836b9e-6c89-43f6-8d02-817c5205c705 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.129718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1311/2639] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 04:59:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 04:59:57.134107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dac204-91f4-4435-9747-ea5a6a72a070 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.144543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dac204-91f4-4435-9747-ea5a6a72a070 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.149566 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab4dbe4-2695-49f7-b50f-ec2994cecd45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.150272 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab4dbe4-2695-49f7-b50f-ec2994cecd45 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.150552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab4dbe4-2695-49f7-b50f-ec2994cecd45 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.150835 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab4dbe4-2695-49f7-b50f-ec2994cecd45 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.151506 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab4dbe4-2695-49f7-b50f-ec2994cecd45 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.153514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab4dbe4-2695-49f7-b50f-ec2994cecd45 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.180970 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1329/2640] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:59:57 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:57.185948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84836b9e-6c89-43f6-8d02-817c5205c705 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.196293 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84836b9e-6c89-43f6-8d02-817c5205c705 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.252825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae934fe8-a44a-4d31-84bb-d1b7b49649e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.253449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae934fe8-a44a-4d31-84bb-d1b7b49649e4 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.253734 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae934fe8-a44a-4d31-84bb-d1b7b49649e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.254015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae934fe8-a44a-4d31-84bb-d1b7b49649e4 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.254686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae934fe8-a44a-4d31-84bb-d1b7b49649e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.256635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae934fe8-a44a-4d31-84bb-d1b7b49649e4 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.276776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1312/2641] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:59:57 2022] POST /identity/v3/users => generated 288 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:57.281020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab4dbe4-2695-49f7-b50f-ec2994cecd45 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.286560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:57.292354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab4dbe4-2695-49f7-b50f-ec2994cecd45 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.298462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de956d7-9b85-4ae5-830a-0091ff806a3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.299488 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de956d7-9b85-4ae5-830a-0091ff806a3b admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.299784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de956d7-9b85-4ae5-830a-0091ff806a3b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.300065 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de956d7-9b85-4ae5-830a-0091ff806a3b admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.300811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de956d7-9b85-4ae5-830a-0091ff806a3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.302450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de956d7-9b85-4ae5-830a-0091ff806a3b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.313068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1330/2642] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:57.316943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae934fe8-a44a-4d31-84bb-d1b7b49649e4 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.327720 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae934fe8-a44a-4d31-84bb-d1b7b49649e4 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.332406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c73694-8650-41f1-bc20-6a4a567a5a0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.333058 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c73694-8650-41f1-bc20-6a4a567a5a0f admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.333344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c73694-8650-41f1-bc20-6a4a567a5a0f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.333632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c73694-8650-41f1-bc20-6a4a567a5a0f admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.334325 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c73694-8650-41f1-bc20-6a4a567a5a0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.335913 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c73694-8650-41f1-bc20-6a4a567a5a0f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.346687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1313/2643] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:57.350582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de956d7-9b85-4ae5-830a-0091ff806a3b admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.360725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de956d7-9b85-4ae5-830a-0091ff806a3b admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.365353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7a4a45-0c7b-4edf-80a1-ece060040e7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.366010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7a4a45-0c7b-4edf-80a1-ece060040e7e admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.366299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7a4a45-0c7b-4edf-80a1-ece060040e7e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.366584 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7a4a45-0c7b-4edf-80a1-ece060040e7e admin admin] PATH_INFO: `/v3/projects/adf1f4a0997d48c4848a3aa93fa2961b/users/f4149c058a184945b23627c0ae2568f0/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.381612 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7a4a45-0c7b-4edf-80a1-ece060040e7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=adf1f4a0997d48c4848a3aa93fa2961b, user_id=f4149c058a184945b23627c0ae2568f0, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.384252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e7a4a45-0c7b-4edf-80a1-ece060040e7e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.397578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1331/2644] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:57 2022] PUT /identity/v3/projects/adf1f4a0997d48c4848a3aa93fa2961b/users/f4149c058a184945b23627c0ae2568f0/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:57.408081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109fc46c-c35a-4834-aa84-491a06cbf8ea None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.408371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109fc46c-c35a-4834-aa84-491a06cbf8ea None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.408673 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109fc46c-c35a-4834-aa84-491a06cbf8ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.437576 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-109fc46c-c35a-4834-aa84-491a06cbf8ea None None] MFA Rules not processed for user `f4149c058a184945b23627c0ae2568f0`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:57.438810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-109fc46c-c35a-4834-aa84-491a06cbf8ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:57.501792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1314/2645] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:57 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:57.506325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7a4a45-0c7b-4edf-80a1-ece060040e7e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.568160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7a4a45-0c7b-4edf-80a1-ece060040e7e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.574908 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe7e9bf-ae12-4272-8bfb-8ad43d566ea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.575533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe7e9bf-ae12-4272-8bfb-8ad43d566ea4 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.575840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe7e9bf-ae12-4272-8bfb-8ad43d566ea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.576104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe7e9bf-ae12-4272-8bfb-8ad43d566ea4 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.576806 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe7e9bf-ae12-4272-8bfb-8ad43d566ea4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.578708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe7e9bf-ae12-4272-8bfb-8ad43d566ea4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.605341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1332/2646] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 04:59:57 2022] POST /identity/v3/projects => generated 373 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:57.609089 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109fc46c-c35a-4834-aa84-491a06cbf8ea None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.621064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109fc46c-c35a-4834-aa84-491a06cbf8ea None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.635683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:57.689098 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc84b3a-435f-4a57-a9e8-377d27fcb9c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.689703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc84b3a-435f-4a57-a9e8-377d27fcb9c9 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.689986 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc84b3a-435f-4a57-a9e8-377d27fcb9c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.690261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc84b3a-435f-4a57-a9e8-377d27fcb9c9 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.690918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc84b3a-435f-4a57-a9e8-377d27fcb9c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.692842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc84b3a-435f-4a57-a9e8-377d27fcb9c9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.712336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1315/2647] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 04:59:57 2022] POST /identity/v3/users => generated 293 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 04:59:57.716375 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe7e9bf-ae12-4272-8bfb-8ad43d566ea4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.726516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe7e9bf-ae12-4272-8bfb-8ad43d566ea4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.731767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23acd0b-1d9e-4082-9616-719713676660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.732376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23acd0b-1d9e-4082-9616-719713676660 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.732726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23acd0b-1d9e-4082-9616-719713676660 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.733009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23acd0b-1d9e-4082-9616-719713676660 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.733705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23acd0b-1d9e-4082-9616-719713676660 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.735304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23acd0b-1d9e-4082-9616-719713676660 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.746644 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1333/2648] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 04:59:57.750519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc84b3a-435f-4a57-a9e8-377d27fcb9c9 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.760349 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc84b3a-435f-4a57-a9e8-377d27fcb9c9 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.764969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1192ab-6547-4b0e-8598-637d41b20505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.765636 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1192ab-6547-4b0e-8598-637d41b20505 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.765903 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1192ab-6547-4b0e-8598-637d41b20505 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.766184 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1192ab-6547-4b0e-8598-637d41b20505 admin admin] PATH_INFO: `/v3/projects/3d80d46f72d045fd8796222fb26e0ec4/users/27eff0aae72f468e8e0403396214ad3f/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.781146 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1192ab-6547-4b0e-8598-637d41b20505 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d80d46f72d045fd8796222fb26e0ec4, user_id=27eff0aae72f468e8e0403396214ad3f, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.783787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1192ab-6547-4b0e-8598-637d41b20505 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.797616 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1316/2649] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:57 2022] PUT /identity/v3/projects/3d80d46f72d045fd8796222fb26e0ec4/users/27eff0aae72f468e8e0403396214ad3f/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 04:59:57.801625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23acd0b-1d9e-4082-9616-719713676660 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.811645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23acd0b-1d9e-4082-9616-719713676660 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.867422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30321aa5-81fe-4974-92da-1eb38e8abc4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.868049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30321aa5-81fe-4974-92da-1eb38e8abc4d admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.868333 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30321aa5-81fe-4974-92da-1eb38e8abc4d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.868626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30321aa5-81fe-4974-92da-1eb38e8abc4d admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.869327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30321aa5-81fe-4974-92da-1eb38e8abc4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.870939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30321aa5-81fe-4974-92da-1eb38e8abc4d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.881028 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1334/2650] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 04:59:57 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 04:59:57.885263 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1192ab-6547-4b0e-8598-637d41b20505 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:57.895580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1192ab-6547-4b0e-8598-637d41b20505 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:57.900846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd6b1f1-f031-470d-9541-626bb84bcbb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:57.901500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd6b1f1-f031-470d-9541-626bb84bcbb1 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.901788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd6b1f1-f031-470d-9541-626bb84bcbb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.902071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd6b1f1-f031-470d-9541-626bb84bcbb1 admin admin] PATH_INFO: `/v3/projects/3d80d46f72d045fd8796222fb26e0ec4/users/27eff0aae72f468e8e0403396214ad3f/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:57.904051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd6b1f1-f031-470d-9541-626bb84bcbb1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d80d46f72d045fd8796222fb26e0ec4, user_id=27eff0aae72f468e8e0403396214ad3f, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:57.906788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd6b1f1-f031-470d-9541-626bb84bcbb1 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:57.917469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1317/2651] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 04:59:57 2022] PUT /identity/v3/projects/3d80d46f72d045fd8796222fb26e0ec4/users/27eff0aae72f468e8e0403396214ad3f/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 04:59:57.983274 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe2ae60-70de-47fd-8b32-bbfa04595d93 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:57.983555 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe2ae60-70de-47fd-8b32-bbfa04595d93 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:57.983979 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe2ae60-70de-47fd-8b32-bbfa04595d93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:58.012341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-dfe2ae60-70de-47fd-8b32-bbfa04595d93 None None] MFA Rules not processed for user `27eff0aae72f468e8e0403396214ad3f`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 04:59:58.013593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-dfe2ae60-70de-47fd-8b32-bbfa04595d93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:58.039507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 04:59:58.083106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1335/2652] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 04:59:57 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 04:59:58.093843 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd6b1f1-f031-470d-9541-626bb84bcbb1 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 04:59:58.095676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-edd6b1f1-f031-470d-9541-626bb84bcbb1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 04:59:58.161209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd6b1f1-f031-470d-9541-626bb84bcbb1 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 04:59:58.165001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e49029-726a-4326-8f87-ac306ce33c76 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 04:59:58.165627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e49029-726a-4326-8f87-ac306ce33c76 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 04:59:58.166521 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e49029-726a-4326-8f87-ac306ce33c76 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 04:59:58.166856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e49029-726a-4326-8f87-ac306ce33c76 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 04:59:58.221449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e49029-726a-4326-8f87-ac306ce33c76 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 04:59:58.222885 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e49029-726a-4326-8f87-ac306ce33c76 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 04:59:58.232194 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1318/2653] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 04:59:58 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:00.251027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe2ae60-70de-47fd-8b32-bbfa04595d93 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:00.256790 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-dfe2ae60-70de-47fd-8b32-bbfa04595d93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:00.328315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe2ae60-70de-47fd-8b32-bbfa04595d93 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:00.332912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:00.333562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:00.333854 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:00.334138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] PATH_INFO: `/v3/users/7136bb1746744ce89f74c1185c76d043` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:00.335351 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7136bb1746744ce89f74c1185c76d043)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:00.337382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:00.389525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7136bb1746744ce89f74c1185c76d043'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.390190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7136bb1746744ce89f74c1185c76d043'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.390935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7136bb1746744ce89f74c1185c76d043'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.406987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7136bb1746744ce89f74c1185c76d043'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.414596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7136bb1746744ce89f74c1185c76d043'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.428553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1336/2654] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:00 2022] DELETE /identity/v3/users/7136bb1746744ce89f74c1185c76d043 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:00.603706 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e49029-726a-4326-8f87-ac306ce33c76 service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:00.617483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e49029-726a-4326-8f87-ac306ce33c76 service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:00.642696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:00.684213 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:00.684902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:00.685229 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:00.685530 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] PATH_INFO: `/v3/projects/62319662d8e34fc8bbe01e0410cec30a` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:00.686682 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] RBAC: Authorizing `identity:delete_project(project_id=62319662d8e34fc8bbe01e0410cec30a)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:00.688674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:00.734850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62319662d8e34fc8bbe01e0410cec30a', 'request_id': 'req-688d4742-734c-43d9-9f5b-f40865059433'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.742751 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62319662d8e34fc8bbe01e0410cec30a', 'request_id': 'req-688d4742-734c-43d9-9f5b-f40865059433'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.749560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62319662d8e34fc8bbe01e0410cec30a', 'request_id': 'req-688d4742-734c-43d9-9f5b-f40865059433'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.761602 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:00.763212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:00.764354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] The token cache is being invalidate because project 62319662d8e34fc8bbe01e0410cec30a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:00.764769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.765421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-688d4742-734c-43d9-9f5b-f40865059433 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.766940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1319/2655] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:00 2022] DELETE /identity/v3/projects/62319662d8e34fc8bbe01e0410cec30a => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:00.777479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd2883b-04dc-4e98-93e4-b914bbccf7b0 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:00.823825 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:00.843744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:00.844421 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:00.844762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:00.845049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] PATH_INFO: `/v3/users/f4149c058a184945b23627c0ae2568f0` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:00.846334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f4149c058a184945b23627c0ae2568f0)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:00.848383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:00.900464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f4149c058a184945b23627c0ae2568f0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.901732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f4149c058a184945b23627c0ae2568f0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.902361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f4149c058a184945b23627c0ae2568f0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.915438 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f4149c058a184945b23627c0ae2568f0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.923589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f4149c058a184945b23627c0ae2568f0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:00.938441 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1337/2656] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:00 2022] DELETE /identity/v3/users/f4149c058a184945b23627c0ae2568f0 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:00.942954 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd3b4c4-1e8e-4de4-8b9c-a5fc9cb19df7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:00.943246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd3b4c4-1e8e-4de4-8b9c-a5fc9cb19df7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:00.943543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd3b4c4-1e8e-4de4-8b9c-a5fc9cb19df7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:00.965551 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-bcd3b4c4-1e8e-4de4-8b9c-a5fc9cb19df7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:00.966864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-bcd3b4c4-1e8e-4de4-8b9c-a5fc9cb19df7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:01.033094 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1320/2657] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:01.043465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:01.044944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:01.102095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61adee61-fec6-48ca-a85c-05918bce3e35 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:01.106611 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f98bd05-8fe4-43ee-9a62-3ce6ccbf86af service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:01.107236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f98bd05-8fe4-43ee-9a62-3ce6ccbf86af service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:01.107526 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f98bd05-8fe4-43ee-9a62-3ce6ccbf86af service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:01.107811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f98bd05-8fe4-43ee-9a62-3ce6ccbf86af service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:01.116781 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f98bd05-8fe4-43ee-9a62-3ce6ccbf86af service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:01.118219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f98bd05-8fe4-43ee-9a62-3ce6ccbf86af service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:01.127935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1338/2658] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:01.388150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd3b4c4-1e8e-4de4-8b9c-a5fc9cb19df7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:01.389671 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-bcd3b4c4-1e8e-4de4-8b9c-a5fc9cb19df7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:01.402987 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd3b4c4-1e8e-4de4-8b9c-a5fc9cb19df7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:01.407717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:01.408340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:01.408666 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:01.408952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] PATH_INFO: `/v3/projects/adf1f4a0997d48c4848a3aa93fa2961b` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:01.410126 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] RBAC: Authorizing `identity:delete_project(project_id=adf1f4a0997d48c4848a3aa93fa2961b)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:01.412054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:01.463970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'adf1f4a0997d48c4848a3aa93fa2961b', 'request_id': 'req-acf4f79d-2251-43b1-8a47-6ef972867612'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:01.465782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:01.472370 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'adf1f4a0997d48c4848a3aa93fa2961b', 'request_id': 'req-acf4f79d-2251-43b1-8a47-6ef972867612'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:01.479107 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'adf1f4a0997d48c4848a3aa93fa2961b', 'request_id': 'req-acf4f79d-2251-43b1-8a47-6ef972867612'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:01.491226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] The token cache is being invalidate because project adf1f4a0997d48c4848a3aa93fa2961b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:01.491522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:01.492282 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:01.494065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1321/2659] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:01 2022] DELETE /identity/v3/projects/adf1f4a0997d48c4848a3aa93fa2961b => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:01.499145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f98bd05-8fe4-43ee-9a62-3ce6ccbf86af service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:01.509507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f98bd05-8fe4-43ee-9a62-3ce6ccbf86af service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:01.571143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:01.571795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:01.572086 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:01.572369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] PATH_INFO: `/v3/users/27eff0aae72f468e8e0403396214ad3f` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:01.574192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] RBAC: Authorizing `identity:delete_user(user_id=27eff0aae72f468e8e0403396214ad3f)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:01.576110 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:01.580674 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:01.630743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '27eff0aae72f468e8e0403396214ad3f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:01.631403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '27eff0aae72f468e8e0403396214ad3f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:01.632013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '27eff0aae72f468e8e0403396214ad3f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:01.644431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '27eff0aae72f468e8e0403396214ad3f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:01.651361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-439bb646-23de-4ef0-b89c-1dec82100672 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '27eff0aae72f468e8e0403396214ad3f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:01.664462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1339/2660] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:01 2022] DELETE /identity/v3/users/27eff0aae72f468e8e0403396214ad3f => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:01.944816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:01.946314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:01.959727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf4f79d-2251-43b1-8a47-6ef972867612 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:02.016540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:02.017143 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.017430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.017718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] PATH_INFO: `/v3/projects/3d80d46f72d045fd8796222fb26e0ec4` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.018882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d80d46f72d045fd8796222fb26e0ec4)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:02.020838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:02.067046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d80d46f72d045fd8796222fb26e0ec4', 'request_id': 'req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:02.074510 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d80d46f72d045fd8796222fb26e0ec4', 'request_id': 'req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:02.079980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d80d46f72d045fd8796222fb26e0ec4', 'request_id': 'req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:02.091723 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] The token cache is being invalidate because project 3d80d46f72d045fd8796222fb26e0ec4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:02.092015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:02.092683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-f7f90814-c9df-47b8-aa9f-5ebed2851fb6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:02.094469 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1322/2661] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:01 2022] DELETE /identity/v3/projects/3d80d46f72d045fd8796222fb26e0ec4 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:02.099561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b23897-6fc1-4dd3-adbb-6c7be209c110 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.099855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b23897-6fc1-4dd3-adbb-6c7be209c110 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.100139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b23897-6fc1-4dd3-adbb-6c7be209c110 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.120529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-08b23897-6fc1-4dd3-adbb-6c7be209c110 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:02.121794 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-08b23897-6fc1-4dd3-adbb-6c7be209c110 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:02.181189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1340/2662] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:02.189499 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1921505-9f50-408c-a8c4-1462d61f7802 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.189796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1921505-9f50-408c-a8c4-1462d61f7802 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.190084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1921505-9f50-408c-a8c4-1462d61f7802 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.209856 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-d1921505-9f50-408c-a8c4-1462d61f7802 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:02.211069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d1921505-9f50-408c-a8c4-1462d61f7802 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:02.219120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1323/2663] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 05:00:02.223266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b23897-6fc1-4dd3-adbb-6c7be209c110 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:02.233613 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b23897-6fc1-4dd3-adbb-6c7be209c110 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:02.238095 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26613272-5317-42bb-b197-e38c487c0690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:02.238700 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26613272-5317-42bb-b197-e38c487c0690 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.238985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26613272-5317-42bb-b197-e38c487c0690 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.239262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26613272-5317-42bb-b197-e38c487c0690 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.239965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26613272-5317-42bb-b197-e38c487c0690 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 05:00:02.240270 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26613272-5317-42bb-b197-e38c487c0690 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:02.241935 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26613272-5317-42bb-b197-e38c487c0690 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:02.253906 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1341/2664] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 05:00:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 05:00:02.258082 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1921505-9f50-408c-a8c4-1462d61f7802 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:02.267957 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1921505-9f50-408c-a8c4-1462d61f7802 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:02.272543 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da8f98c-44ce-42fc-a3a2-22ee34faedea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:02.273193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da8f98c-44ce-42fc-a3a2-22ee34faedea admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.273476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da8f98c-44ce-42fc-a3a2-22ee34faedea admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.273752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da8f98c-44ce-42fc-a3a2-22ee34faedea admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.274412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da8f98c-44ce-42fc-a3a2-22ee34faedea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:02.277498 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da8f98c-44ce-42fc-a3a2-22ee34faedea admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:02.304849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1324/2665] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:02 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:02.309153 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26613272-5317-42bb-b197-e38c487c0690 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:02.319089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26613272-5317-42bb-b197-e38c487c0690 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:02.374983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a51450-3a6e-4d1a-83c0-f32c2c1cb7d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:02.375618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a51450-3a6e-4d1a-83c0-f32c2c1cb7d8 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.375912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a51450-3a6e-4d1a-83c0-f32c2c1cb7d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.376196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a51450-3a6e-4d1a-83c0-f32c2c1cb7d8 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.376896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a51450-3a6e-4d1a-83c0-f32c2c1cb7d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:02.378833 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a51450-3a6e-4d1a-83c0-f32c2c1cb7d8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:02.395219 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:02.402101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1342/2666] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:02 2022] POST /identity/v3/users => generated 286 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:02.405546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da8f98c-44ce-42fc-a3a2-22ee34faedea admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:02.416509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da8f98c-44ce-42fc-a3a2-22ee34faedea admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:02.422032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c48353b-9945-430d-9ea4-d0dc0ff1f0db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:02.422645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c48353b-9945-430d-9ea4-d0dc0ff1f0db admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.422934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c48353b-9945-430d-9ea4-d0dc0ff1f0db admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.423219 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c48353b-9945-430d-9ea4-d0dc0ff1f0db admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.423918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c48353b-9945-430d-9ea4-d0dc0ff1f0db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:02.425556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c48353b-9945-430d-9ea4-d0dc0ff1f0db admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:02.435458 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1325/2667] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:02.439308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a51450-3a6e-4d1a-83c0-f32c2c1cb7d8 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:02.449465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a51450-3a6e-4d1a-83c0-f32c2c1cb7d8 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:02.454120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875221ee-e54f-4203-8cd0-30fbe4242a7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:02.454733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875221ee-e54f-4203-8cd0-30fbe4242a7e admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.455023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875221ee-e54f-4203-8cd0-30fbe4242a7e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.455305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875221ee-e54f-4203-8cd0-30fbe4242a7e admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.456000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875221ee-e54f-4203-8cd0-30fbe4242a7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:02.457635 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875221ee-e54f-4203-8cd0-30fbe4242a7e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:02.469928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1343/2668] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:02.474213 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c48353b-9945-430d-9ea4-d0dc0ff1f0db admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:02.484980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c48353b-9945-430d-9ea4-d0dc0ff1f0db admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:02.489598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c9ddfc-b6f3-4f1f-a68d-f247d1194edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:02.490258 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9ddfc-b6f3-4f1f-a68d-f247d1194edb admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.490550 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9ddfc-b6f3-4f1f-a68d-f247d1194edb admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.490832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9ddfc-b6f3-4f1f-a68d-f247d1194edb admin admin] PATH_INFO: `/v3/projects/9b4791b868c04418be1ba2c79ab278a9/users/77ef2b72f9224ebab5340cff47c7eb5b/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.505740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c9ddfc-b6f3-4f1f-a68d-f247d1194edb admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b4791b868c04418be1ba2c79ab278a9, user_id=77ef2b72f9224ebab5340cff47c7eb5b, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:02.508394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c9ddfc-b6f3-4f1f-a68d-f247d1194edb admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:02.521040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1326/2669] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:02 2022] PUT /identity/v3/projects/9b4791b868c04418be1ba2c79ab278a9/users/77ef2b72f9224ebab5340cff47c7eb5b/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:02.532117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156e17c6-07c4-48e4-b5e0-9c159c24b42d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.532589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156e17c6-07c4-48e4-b5e0-9c159c24b42d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.532723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156e17c6-07c4-48e4-b5e0-9c159c24b42d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.561723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-156e17c6-07c4-48e4-b5e0-9c159c24b42d None None] MFA Rules not processed for user `77ef2b72f9224ebab5340cff47c7eb5b`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:02.562953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-156e17c6-07c4-48e4-b5e0-9c159c24b42d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:02.632745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1344/2670] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:02 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:02.645759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c9ddfc-b6f3-4f1f-a68d-f247d1194edb admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:02.647476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-85c9ddfc-b6f3-4f1f-a68d-f247d1194edb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:02.658129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:INSERT=4,keystone:DELETE=10,keystone:SELECT=86 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:02.717585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c9ddfc-b6f3-4f1f-a68d-f247d1194edb admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:02.721235 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fe6ab2-5502-4934-af30-a6ae8ef6ae4d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3749dc021aac478f88072ced48f4cee4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:02.721847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fe6ab2-5502-4934-af30-a6ae8ef6ae4d service glance] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:02.722130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fe6ab2-5502-4934-af30-a6ae8ef6ae4d service glance] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:02.722420 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fe6ab2-5502-4934-af30-a6ae8ef6ae4d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:02.730891 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fe6ab2-5502-4934-af30-a6ae8ef6ae4d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:02.732322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fe6ab2-5502-4934-af30-a6ae8ef6ae4d service glance] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:02.741151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1327/2671] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:02 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:04.639489 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156e17c6-07c4-48e4-b5e0-9c159c24b42d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:04.640944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-156e17c6-07c4-48e4-b5e0-9c159c24b42d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:04.699717 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156e17c6-07c4-48e4-b5e0-9c159c24b42d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:04.703341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b41e2e-0083-409b-bf8c-9124b927635a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:04.703960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b41e2e-0083-409b-bf8c-9124b927635a service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:04.704251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b41e2e-0083-409b-bf8c-9124b927635a service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:04.704532 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b41e2e-0083-409b-bf8c-9124b927635a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:04.706331 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-43b41e2e-0083-409b-bf8c-9124b927635a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:04.789730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b41e2e-0083-409b-bf8c-9124b927635a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:04.791168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b41e2e-0083-409b-bf8c-9124b927635a service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:04.811912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1345/2672] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:04 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:04.863708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fe6ab2-5502-4934-af30-a6ae8ef6ae4d service glance] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:04.865309 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-95fe6ab2-5502-4934-af30-a6ae8ef6ae4d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:04.941185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fe6ab2-5502-4934-af30-a6ae8ef6ae4d service glance] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:04.945808 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:04.946447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:04.946731 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:04.947013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] PATH_INFO: `/v3/users/ca007ca744e9499cb236aa4e7831d1cc` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:04.948230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca007ca744e9499cb236aa4e7831d1cc)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:04.950494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:05.003769 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca007ca744e9499cb236aa4e7831d1cc'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:05.012002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca007ca744e9499cb236aa4e7831d1cc'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:05.012821 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca007ca744e9499cb236aa4e7831d1cc'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:05.013527 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca007ca744e9499cb236aa4e7831d1cc'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:05.021427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca007ca744e9499cb236aa4e7831d1cc'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:05.039780 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1328/2673] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:04 2022] DELETE /identity/v3/users/ca007ca744e9499cb236aa4e7831d1cc => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:05.044736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee55119c-c934-4dfa-ba9f-fb41fd3b952d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:05.045162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee55119c-c934-4dfa-ba9f-fb41fd3b952d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:05.045536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee55119c-c934-4dfa-ba9f-fb41fd3b952d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:05.054278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:05.066741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-ee55119c-c934-4dfa-ba9f-fb41fd3b952d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:05.068131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ee55119c-c934-4dfa-ba9f-fb41fd3b952d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:05.134936 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1346/2674] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 05:00:05.145260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:05.146678 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:05.185883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:05.218870 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e08aa61-1251-4f02-b34f-49681298bb74 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:05.222907 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c1afeb-dce9-4104-9540-5f492c19c8f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:05.223537 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c1afeb-dce9-4104-9540-5f492c19c8f2 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:05.223825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c1afeb-dce9-4104-9540-5f492c19c8f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:05.224106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c1afeb-dce9-4104-9540-5f492c19c8f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:05.235659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c1afeb-dce9-4104-9540-5f492c19c8f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:05.237495 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c1afeb-dce9-4104-9540-5f492c19c8f2 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:05.249454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1329/2675] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:05.535510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee55119c-c934-4dfa-ba9f-fb41fd3b952d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:05.537021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ee55119c-c934-4dfa-ba9f-fb41fd3b952d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:05.549852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee55119c-c934-4dfa-ba9f-fb41fd3b952d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:05.554568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:05.555215 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:05.555502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:05.555784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] PATH_INFO: `/v3/projects/e5a73fe26f134102bf88e5e06da0866c` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:05.557131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e5a73fe26f134102bf88e5e06da0866c)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:05.560395 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:05.610049 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e5a73fe26f134102bf88e5e06da0866c', 'request_id': 'req-9655aa44-a7f7-4be8-860b-740948d57827'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:05.616557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e5a73fe26f134102bf88e5e06da0866c', 'request_id': 'req-9655aa44-a7f7-4be8-860b-740948d57827'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:05.624842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e5a73fe26f134102bf88e5e06da0866c', 'request_id': 'req-9655aa44-a7f7-4be8-860b-740948d57827'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:05.635170 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] The token cache is being invalidate because project e5a73fe26f134102bf88e5e06da0866c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:05.635457 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:05.636072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-9655aa44-a7f7-4be8-860b-740948d57827 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:05.641094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1347/2676] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:05 2022] DELETE /identity/v3/projects/e5a73fe26f134102bf88e5e06da0866c => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:05.642783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd644460-a1d6-4fad-a5cc-2bb6aa19c9c4 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:05.643074 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd644460-a1d6-4fad-a5cc-2bb6aa19c9c4 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:05.643354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd644460-a1d6-4fad-a5cc-2bb6aa19c9c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:05.669006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-dd644460-a1d6-4fad-a5cc-2bb6aa19c9c4 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:05.670233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-dd644460-a1d6-4fad-a5cc-2bb6aa19c9c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:05.736777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1330/2677] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:05.744593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5e98f5-1eac-4b20-b3b7-924f2f657b5a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:05.744909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5e98f5-1eac-4b20-b3b7-924f2f657b5a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:05.745198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5e98f5-1eac-4b20-b3b7-924f2f657b5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:05.765917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-1f5e98f5-1eac-4b20-b3b7-924f2f657b5a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:05.767156 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-1f5e98f5-1eac-4b20-b3b7-924f2f657b5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:05.775481 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1348/2678] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:05.780621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd644460-a1d6-4fad-a5cc-2bb6aa19c9c4 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:05.790687 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd644460-a1d6-4fad-a5cc-2bb6aa19c9c4 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:05.795916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6536e1-6fd8-47dd-b0bd-4c2e2afd3768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:05.796532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6536e1-6fd8-47dd-b0bd-4c2e2afd3768 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:05.796850 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6536e1-6fd8-47dd-b0bd-4c2e2afd3768 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:05.797156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6536e1-6fd8-47dd-b0bd-4c2e2afd3768 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:05.797858 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6536e1-6fd8-47dd-b0bd-4c2e2afd3768 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 05:00:05.798166 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6536e1-6fd8-47dd-b0bd-4c2e2afd3768 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:05.799781 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6536e1-6fd8-47dd-b0bd-4c2e2afd3768 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:05.812627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1331/2679] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 05:00:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 05:00:05.816989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5e98f5-1eac-4b20-b3b7-924f2f657b5a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:05.827199 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5e98f5-1eac-4b20-b3b7-924f2f657b5a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:05.831894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c46f7c6-d4ed-42b3-ad5f-46edafbdab56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:05.832507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c46f7c6-d4ed-42b3-ad5f-46edafbdab56 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:05.832823 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c46f7c6-d4ed-42b3-ad5f-46edafbdab56 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:05.833103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c46f7c6-d4ed-42b3-ad5f-46edafbdab56 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:05.833775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c46f7c6-d4ed-42b3-ad5f-46edafbdab56 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:05.835680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c46f7c6-d4ed-42b3-ad5f-46edafbdab56 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:05.864315 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1349/2680] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:05 2022] POST /identity/v3/projects => generated 371 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:05.868786 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6536e1-6fd8-47dd-b0bd-4c2e2afd3768 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:05.879534 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6536e1-6fd8-47dd-b0bd-4c2e2afd3768 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:06.068267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf195fec-0dea-4ec8-86c3-115407f1f45e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:06.068845 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf195fec-0dea-4ec8-86c3-115407f1f45e admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:06.069160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf195fec-0dea-4ec8-86c3-115407f1f45e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:06.069461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf195fec-0dea-4ec8-86c3-115407f1f45e admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:06.070138 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf195fec-0dea-4ec8-86c3-115407f1f45e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:06.072037 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf195fec-0dea-4ec8-86c3-115407f1f45e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:06.096306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1332/2681] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:05 2022] POST /identity/v3/users => generated 286 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:00:06.097811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c46f7c6-d4ed-42b3-ad5f-46edafbdab56 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:06.110133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c46f7c6-d4ed-42b3-ad5f-46edafbdab56 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:06.114883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff06dd0f-af00-4024-9a35-610c320595f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:06.115514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff06dd0f-af00-4024-9a35-610c320595f9 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:06.115802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff06dd0f-af00-4024-9a35-610c320595f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:06.116081 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff06dd0f-af00-4024-9a35-610c320595f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:06.116815 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff06dd0f-af00-4024-9a35-610c320595f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:06.118431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff06dd0f-af00-4024-9a35-610c320595f9 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:06.129224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1350/2682] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:06 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:06.133341 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf195fec-0dea-4ec8-86c3-115407f1f45e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:06.144348 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf195fec-0dea-4ec8-86c3-115407f1f45e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:06.149111 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84eba3a-9740-4f5d-8127-8b82a7a82f1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:06.149740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84eba3a-9740-4f5d-8127-8b82a7a82f1f admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:06.150034 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84eba3a-9740-4f5d-8127-8b82a7a82f1f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:06.150317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84eba3a-9740-4f5d-8127-8b82a7a82f1f admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:06.151023 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84eba3a-9740-4f5d-8127-8b82a7a82f1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:06.153289 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84eba3a-9740-4f5d-8127-8b82a7a82f1f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:06.163767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1333/2683] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:06 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:06.168345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff06dd0f-af00-4024-9a35-610c320595f9 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:06.178419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff06dd0f-af00-4024-9a35-610c320595f9 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:06.182973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fc66f9-fb2c-4d05-a0ef-79a52836ac12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:06.184187 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fc66f9-fb2c-4d05-a0ef-79a52836ac12 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:06.184473 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fc66f9-fb2c-4d05-a0ef-79a52836ac12 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:06.184797 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fc66f9-fb2c-4d05-a0ef-79a52836ac12 admin admin] PATH_INFO: `/v3/projects/1d8ff0272aa64b32bfcb3f164f258f50/users/b1d4e0f8a1f14aeca23027c254d11042/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:06.201113 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fc66f9-fb2c-4d05-a0ef-79a52836ac12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d8ff0272aa64b32bfcb3f164f258f50, user_id=b1d4e0f8a1f14aeca23027c254d11042, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:06.203770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fc66f9-fb2c-4d05-a0ef-79a52836ac12 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:06.217304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1351/2684] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:06 2022] PUT /identity/v3/projects/1d8ff0272aa64b32bfcb3f164f258f50/users/b1d4e0f8a1f14aeca23027c254d11042/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:06.229424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74020681-a1c2-4b2f-8a4f-c4fffc40b865 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:06.229710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74020681-a1c2-4b2f-8a4f-c4fffc40b865 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:06.229988 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74020681-a1c2-4b2f-8a4f-c4fffc40b865 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:06.258657 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-74020681-a1c2-4b2f-8a4f-c4fffc40b865 None None] MFA Rules not processed for user `b1d4e0f8a1f14aeca23027c254d11042`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:06.259877 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-74020681-a1c2-4b2f-8a4f-c4fffc40b865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:06.283818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:06.335883 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1334/2685] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:06 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:06.346878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fc66f9-fb2c-4d05-a0ef-79a52836ac12 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:06.354023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-15fc66f9-fb2c-4d05-a0ef-79a52836ac12 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:06.388562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:06.424895 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fc66f9-fb2c-4d05-a0ef-79a52836ac12 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:06.428912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa8e48a-4e45-46a0-a407-acc53e841b00 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:06.429536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa8e48a-4e45-46a0-a407-acc53e841b00 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:06.429820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa8e48a-4e45-46a0-a407-acc53e841b00 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:06.430097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa8e48a-4e45-46a0-a407-acc53e841b00 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:06.439577 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa8e48a-4e45-46a0-a407-acc53e841b00 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:06.441036 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa8e48a-4e45-46a0-a407-acc53e841b00 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:06.450754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1352/2686] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:06 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:06.482317 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74020681-a1c2-4b2f-8a4f-c4fffc40b865 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:06.549621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74020681-a1c2-4b2f-8a4f-c4fffc40b865 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:06.554075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:06.554711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:06.554991 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:06.555268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] PATH_INFO: `/v3/users/b1d4e0f8a1f14aeca23027c254d11042` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:06.556424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1d4e0f8a1f14aeca23027c254d11042)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:06.558393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:06.607725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1d4e0f8a1f14aeca23027c254d11042'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:06.614758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1d4e0f8a1f14aeca23027c254d11042'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:06.615383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1d4e0f8a1f14aeca23027c254d11042'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:06.615977 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1d4e0f8a1f14aeca23027c254d11042'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:06.624110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1d4e0f8a1f14aeca23027c254d11042'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:06.642083 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1335/2687] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:06 2022] DELETE /identity/v3/users/b1d4e0f8a1f14aeca23027c254d11042 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:06.646684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eaeed2-8ae5-4b9f-8f03-c8fc5adb8847 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:06.647189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eaeed2-8ae5-4b9f-8f03-c8fc5adb8847 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:06.647290 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16eaeed2-8ae5-4b9f-8f03-c8fc5adb8847 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:06.670824 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-16eaeed2-8ae5-4b9f-8f03-c8fc5adb8847 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:06.672094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-16eaeed2-8ae5-4b9f-8f03-c8fc5adb8847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:06.731988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1353/2688] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:06.742007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:06.743514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:06.792334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:06.804959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cbfe546-7e85-4b98-a6d6-12ea476f3108 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:06.809099 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917ca2a6-d240-4a97-89cd-a5404eda7a31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:06.809704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917ca2a6-d240-4a97-89cd-a5404eda7a31 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:06.809992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917ca2a6-d240-4a97-89cd-a5404eda7a31 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:06.810295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917ca2a6-d240-4a97-89cd-a5404eda7a31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:06.819436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917ca2a6-d240-4a97-89cd-a5404eda7a31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:06.820908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917ca2a6-d240-4a97-89cd-a5404eda7a31 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:06.830940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1336/2689] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:07.107901 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16eaeed2-8ae5-4b9f-8f03-c8fc5adb8847 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:07.109499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-16eaeed2-8ae5-4b9f-8f03-c8fc5adb8847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:07.122191 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16eaeed2-8ae5-4b9f-8f03-c8fc5adb8847 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:07.127035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:07.127787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:07.128168 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:07.128519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] PATH_INFO: `/v3/projects/1d8ff0272aa64b32bfcb3f164f258f50` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:07.129801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d8ff0272aa64b32bfcb3f164f258f50)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:07.131792 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:07.187784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d8ff0272aa64b32bfcb3f164f258f50', 'request_id': 'req-ef098837-6c01-4ae6-8313-0b5f6e55e12d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:07.194348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d8ff0272aa64b32bfcb3f164f258f50', 'request_id': 'req-ef098837-6c01-4ae6-8313-0b5f6e55e12d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:07.204026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d8ff0272aa64b32bfcb3f164f258f50', 'request_id': 'req-ef098837-6c01-4ae6-8313-0b5f6e55e12d'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:07.215625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] The token cache is being invalidate because project 1d8ff0272aa64b32bfcb3f164f258f50 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:07.216061 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:07.216887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:07.218422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1354/2690] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:07 2022] DELETE /identity/v3/projects/1d8ff0272aa64b32bfcb3f164f258f50 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:12.191368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917ca2a6-d240-4a97-89cd-a5404eda7a31 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:12.200224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917ca2a6-d240-4a97-89cd-a5404eda7a31 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:12.250062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2044295-890b-405c-8f8a-d3591d0912f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:12.250688 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2044295-890b-405c-8f8a-d3591d0912f3 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:12.250973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2044295-890b-405c-8f8a-d3591d0912f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:12.251254 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2044295-890b-405c-8f8a-d3591d0912f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:12.253070 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f2044295-890b-405c-8f8a-d3591d0912f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:12.317718 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2044295-890b-405c-8f8a-d3591d0912f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:12.319163 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2044295-890b-405c-8f8a-d3591d0912f3 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:12.329443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1337/2691] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:15.379602 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:15.382292 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:15.449656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:15.458773 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2044295-890b-405c-8f8a-d3591d0912f3 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:15.460200 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f2044295-890b-405c-8f8a-d3591d0912f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:15.467212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef098837-6c01-4ae6-8313-0b5f6e55e12d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:15.472624 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2044295-890b-405c-8f8a-d3591d0912f3 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:15.474553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dad6a5-7fde-4347-a59a-76834fa38fb4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:15.475301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dad6a5-7fde-4347-a59a-76834fa38fb4 service placement] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:15.475714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dad6a5-7fde-4347-a59a-76834fa38fb4 service placement] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:15.476057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dad6a5-7fde-4347-a59a-76834fa38fb4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:15.477256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:15.477896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:15.478094 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-67dad6a5-7fde-4347-a59a-76834fa38fb4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:15.478230 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:15.478447 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] PATH_INFO: `/v3/users/a1e1c24849d7480b8ac804a7e3d2ab5a` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:15.479622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1e1c24849d7480b8ac804a7e3d2ab5a)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:15.482189 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:15.491463 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dad6a5-7fde-4347-a59a-76834fa38fb4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:15.493078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dad6a5-7fde-4347-a59a-76834fa38fb4 service placement] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:15.503764 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1355/2692] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:15 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:15.531539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1e1c24849d7480b8ac804a7e3d2ab5a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:15.538963 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1e1c24849d7480b8ac804a7e3d2ab5a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:15.539561 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1e1c24849d7480b8ac804a7e3d2ab5a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:15.540197 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1e1c24849d7480b8ac804a7e3d2ab5a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:15.548582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d4d660bd-a71a-40b3-b007-17af8788507d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1e1c24849d7480b8ac804a7e3d2ab5a'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:15.565984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1338/2693] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:15 2022] DELETE /identity/v3/users/a1e1c24849d7480b8ac804a7e3d2ab5a => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:15.739874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dad6a5-7fde-4347-a59a-76834fa38fb4 service placement] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:15.752063 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dad6a5-7fde-4347-a59a-76834fa38fb4 service placement] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:15.815498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:15.816143 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:15.816428 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:15.816743 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] PATH_INFO: `/v3/projects/f8e9e21d168140e99ffed71146e35fc4` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:15.817890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f8e9e21d168140e99ffed71146e35fc4)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:15.819780 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:15.865580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f8e9e21d168140e99ffed71146e35fc4', 'request_id': 'req-ac232705-5d4a-491e-8b6c-e61747df98a4'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:15.872010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f8e9e21d168140e99ffed71146e35fc4', 'request_id': 'req-ac232705-5d4a-491e-8b6c-e61747df98a4'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:15.881448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f8e9e21d168140e99ffed71146e35fc4', 'request_id': 'req-ac232705-5d4a-491e-8b6c-e61747df98a4'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:15.894889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] The token cache is being invalidate because project f8e9e21d168140e99ffed71146e35fc4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:15.894889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:15.894889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-ac232705-5d4a-491e-8b6c-e61747df98a4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:15.895426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1356/2694] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:15 2022] DELETE /identity/v3/projects/f8e9e21d168140e99ffed71146e35fc4 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:15.900401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c5e63f-b6a8-440d-a916-5bb5adce5864 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:15.900971 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c5e63f-b6a8-440d-a916-5bb5adce5864 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:15.901252 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c5e63f-b6a8-440d-a916-5bb5adce5864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:15.926741 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-a2c5e63f-b6a8-440d-a916-5bb5adce5864 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:15.928012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a2c5e63f-b6a8-440d-a916-5bb5adce5864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:15.932300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:15.994084 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1339/2695] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:16.002966 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05852f7f-a32d-4ad2-ad5f-af55a815dd04 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.003267 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05852f7f-a32d-4ad2-ad5f-af55a815dd04 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.003557 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05852f7f-a32d-4ad2-ad5f-af55a815dd04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.024474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-05852f7f-a32d-4ad2-ad5f-af55a815dd04 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:16.025740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-05852f7f-a32d-4ad2-ad5f-af55a815dd04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:16.035044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1357/2696] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:16.039212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c5e63f-b6a8-440d-a916-5bb5adce5864 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:16.049955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c5e63f-b6a8-440d-a916-5bb5adce5864 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:16.054465 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651db479-90b4-495f-b2de-8861a9d57691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:16.055063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651db479-90b4-495f-b2de-8861a9d57691 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.055342 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651db479-90b4-495f-b2de-8861a9d57691 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.055611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651db479-90b4-495f-b2de-8861a9d57691 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.056296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651db479-90b4-495f-b2de-8861a9d57691 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 05:00:16.056617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651db479-90b4-495f-b2de-8861a9d57691 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:16.058205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651db479-90b4-495f-b2de-8861a9d57691 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:16.069980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1340/2697] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 05:00:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 05:00:16.075014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05852f7f-a32d-4ad2-ad5f-af55a815dd04 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:16.084376 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05852f7f-a32d-4ad2-ad5f-af55a815dd04 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:16.091552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa4e5df-013a-4131-9496-d44309085a41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:16.092205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4e5df-013a-4131-9496-d44309085a41 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.092498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4e5df-013a-4131-9496-d44309085a41 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.092821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa4e5df-013a-4131-9496-d44309085a41 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.093503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa4e5df-013a-4131-9496-d44309085a41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:16.095465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa4e5df-013a-4131-9496-d44309085a41 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:16.122172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1358/2698] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:16 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:16.126354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651db479-90b4-495f-b2de-8861a9d57691 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:16.136874 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651db479-90b4-495f-b2de-8861a9d57691 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:16.190597 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5deac0d7-f340-4cbb-bdfb-bd9f2a4353f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:16.191134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deac0d7-f340-4cbb-bdfb-bd9f2a4353f3 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.191421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deac0d7-f340-4cbb-bdfb-bd9f2a4353f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.191695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5deac0d7-f340-4cbb-bdfb-bd9f2a4353f3 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.192362 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5deac0d7-f340-4cbb-bdfb-bd9f2a4353f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:16.194279 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5deac0d7-f340-4cbb-bdfb-bd9f2a4353f3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:16.215573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1341/2699] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:16 2022] POST /identity/v3/users => generated 287 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:16.219636 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa4e5df-013a-4131-9496-d44309085a41 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:16.230098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa4e5df-013a-4131-9496-d44309085a41 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:16.235270 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaf8e78-559c-4ec1-8077-55e8b0f94e3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:16.235893 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf8e78-559c-4ec1-8077-55e8b0f94e3e admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.236183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf8e78-559c-4ec1-8077-55e8b0f94e3e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.236509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eaf8e78-559c-4ec1-8077-55e8b0f94e3e admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.237341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eaf8e78-559c-4ec1-8077-55e8b0f94e3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:16.239068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eaf8e78-559c-4ec1-8077-55e8b0f94e3e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:16.249993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1359/2700] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:16.254015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5deac0d7-f340-4cbb-bdfb-bd9f2a4353f3 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:16.264378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5deac0d7-f340-4cbb-bdfb-bd9f2a4353f3 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:16.269132 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4784ae19-da54-4e5c-a23b-54381c38911b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:16.269736 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4784ae19-da54-4e5c-a23b-54381c38911b admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.270015 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4784ae19-da54-4e5c-a23b-54381c38911b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.270291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4784ae19-da54-4e5c-a23b-54381c38911b admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.270981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4784ae19-da54-4e5c-a23b-54381c38911b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:16.272555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4784ae19-da54-4e5c-a23b-54381c38911b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:16.283872 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1342/2701] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:16.287748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaf8e78-559c-4ec1-8077-55e8b0f94e3e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:16.297881 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eaf8e78-559c-4ec1-8077-55e8b0f94e3e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:16.302572 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcff08f-1301-4c5e-9c7b-556b1ce51253 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:16.303235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcff08f-1301-4c5e-9c7b-556b1ce51253 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.303519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcff08f-1301-4c5e-9c7b-556b1ce51253 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.303799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcff08f-1301-4c5e-9c7b-556b1ce51253 admin admin] PATH_INFO: `/v3/projects/be43f7f8b2524be794f059135e941dd0/users/49e029ab11e849d1a086c05bec640284/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.318239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcff08f-1301-4c5e-9c7b-556b1ce51253 admin admin] RBAC: Authorizing `identity:create_grant(project_id=be43f7f8b2524be794f059135e941dd0, user_id=49e029ab11e849d1a086c05bec640284, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:16.320939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcff08f-1301-4c5e-9c7b-556b1ce51253 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:16.334202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1360/2702] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:16 2022] PUT /identity/v3/projects/be43f7f8b2524be794f059135e941dd0/users/49e029ab11e849d1a086c05bec640284/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:16.345116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9efea03-f9b5-4f65-aa73-fa3d01c117bc None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.345403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9efea03-f9b5-4f65-aa73-fa3d01c117bc None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.345681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9efea03-f9b5-4f65-aa73-fa3d01c117bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.372520 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-d9efea03-f9b5-4f65-aa73-fa3d01c117bc None None] MFA Rules not processed for user `49e029ab11e849d1a086c05bec640284`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:16.373771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d9efea03-f9b5-4f65-aa73-fa3d01c117bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:16.438312 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1343/2703] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:16 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:16.450426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcff08f-1301-4c5e-9c7b-556b1ce51253 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:16.452224 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5dcff08f-1301-4c5e-9c7b-556b1ce51253 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:16.462262 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:16.526213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcff08f-1301-4c5e-9c7b-556b1ce51253 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:16.531875 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e948c0-9d81-4ec1-b58b-90229c08c2bf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:16.532482 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e948c0-9d81-4ec1-b58b-90229c08c2bf service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.532805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e948c0-9d81-4ec1-b58b-90229c08c2bf service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.533090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e948c0-9d81-4ec1-b58b-90229c08c2bf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.540848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e948c0-9d81-4ec1-b58b-90229c08c2bf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:16.542422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e948c0-9d81-4ec1-b58b-90229c08c2bf service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:16.551370 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1361/2704] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:16 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 05:00:16.765765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9efea03-f9b5-4f65-aa73-fa3d01c117bc None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:16.771054 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:16.828492 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9efea03-f9b5-4f65-aa73-fa3d01c117bc None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:16.833002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:16.833637 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.833922 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.834201 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] PATH_INFO: `/v3/users/49e029ab11e849d1a086c05bec640284` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.835355 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] RBAC: Authorizing `identity:delete_user(user_id=49e029ab11e849d1a086c05bec640284)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:16.837302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:16.882691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49e029ab11e849d1a086c05bec640284'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:16.890775 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49e029ab11e849d1a086c05bec640284'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:16.891457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49e029ab11e849d1a086c05bec640284'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:16.892168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49e029ab11e849d1a086c05bec640284'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:16.899534 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49e029ab11e849d1a086c05bec640284'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:16.918233 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1344/2705] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:16 2022] DELETE /identity/v3/users/49e029ab11e849d1a086c05bec640284 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:16.923149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8eaddd-d0ac-47f5-86b3-ddbd3cea3267 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:16.923522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8eaddd-d0ac-47f5-86b3-ddbd3cea3267 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:16.923887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8eaddd-d0ac-47f5-86b3-ddbd3cea3267 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:16.942989 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-3d8eaddd-d0ac-47f5-86b3-ddbd3cea3267 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:16.944206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3d8eaddd-d0ac-47f5-86b3-ddbd3cea3267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:17.004756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1362/2706] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:17.015008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:17.016403 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:17.073640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00fd80a-d317-4947-899a-d3d667fe103f admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:17.077817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b795f2d-71ff-4ba3-bfd4-80bbf3b7a13c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:17.078433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b795f2d-71ff-4ba3-bfd4-80bbf3b7a13c service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:17.078717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b795f2d-71ff-4ba3-bfd4-80bbf3b7a13c service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:17.078993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b795f2d-71ff-4ba3-bfd4-80bbf3b7a13c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:17.088075 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b795f2d-71ff-4ba3-bfd4-80bbf3b7a13c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:17.090709 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b795f2d-71ff-4ba3-bfd4-80bbf3b7a13c service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:17.100430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1345/2707] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 05:00:17.373579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8eaddd-d0ac-47f5-86b3-ddbd3cea3267 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:17.375304 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3d8eaddd-d0ac-47f5-86b3-ddbd3cea3267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:17.388851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8eaddd-d0ac-47f5-86b3-ddbd3cea3267 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:17.393632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:17.394449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:17.394770 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:17.395051 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] PATH_INFO: `/v3/projects/be43f7f8b2524be794f059135e941dd0` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:17.396247 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] RBAC: Authorizing `identity:delete_project(project_id=be43f7f8b2524be794f059135e941dd0)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:17.398202 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:17.576582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be43f7f8b2524be794f059135e941dd0', 'request_id': 'req-a47dda47-387d-433f-971f-46a672b1ec80'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:17.583109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be43f7f8b2524be794f059135e941dd0', 'request_id': 'req-a47dda47-387d-433f-971f-46a672b1ec80'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:17.590453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be43f7f8b2524be794f059135e941dd0', 'request_id': 'req-a47dda47-387d-433f-971f-46a672b1ec80'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:17.600846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] The token cache is being invalidate because project be43f7f8b2524be794f059135e941dd0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:17.601144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:17.601777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a47dda47-387d-433f-971f-46a672b1ec80 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:17.603336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1363/2708] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:17 2022] DELETE /identity/v3/projects/be43f7f8b2524be794f059135e941dd0 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:17.608910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca43715e-e50c-4a1e-b379-a6aae999f17a None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:17.609203 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca43715e-e50c-4a1e-b379-a6aae999f17a None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:17.609482 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca43715e-e50c-4a1e-b379-a6aae999f17a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:17.629155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-ca43715e-e50c-4a1e-b379-a6aae999f17a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:17.630367 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ca43715e-e50c-4a1e-b379-a6aae999f17a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:17.653373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:17.695067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1346/2709] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:17.702903 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7045b024-519c-41a2-b079-12b509c1da8d None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:17.703192 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7045b024-519c-41a2-b079-12b509c1da8d None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:17.703472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7045b024-519c-41a2-b079-12b509c1da8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:17.722829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-7045b024-519c-41a2-b079-12b509c1da8d None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:17.724033 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7045b024-519c-41a2-b079-12b509c1da8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:17.732384 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1364/2710] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:17.736554 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca43715e-e50c-4a1e-b379-a6aae999f17a None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:17.746942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca43715e-e50c-4a1e-b379-a6aae999f17a None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:17.751555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69912bb5-94e2-4dfb-b3bb-4f0402b9aa05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:17.752193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69912bb5-94e2-4dfb-b3bb-4f0402b9aa05 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:17.752480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69912bb5-94e2-4dfb-b3bb-4f0402b9aa05 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:17.752793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69912bb5-94e2-4dfb-b3bb-4f0402b9aa05 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:17.753511 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69912bb5-94e2-4dfb-b3bb-4f0402b9aa05 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 05:00:17.753820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69912bb5-94e2-4dfb-b3bb-4f0402b9aa05 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:17.755442 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69912bb5-94e2-4dfb-b3bb-4f0402b9aa05 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:17.767819 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1347/2711] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 05:00:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 05:00:17.772062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7045b024-519c-41a2-b079-12b509c1da8d None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:17.781816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7045b024-519c-41a2-b079-12b509c1da8d None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:17.786422 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4336aaba-e82c-4298-b971-7818c5d739c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:17.787034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4336aaba-e82c-4298-b971-7818c5d739c8 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:17.787321 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4336aaba-e82c-4298-b971-7818c5d739c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:17.787602 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4336aaba-e82c-4298-b971-7818c5d739c8 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:17.788310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4336aaba-e82c-4298-b971-7818c5d739c8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:17.790245 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4336aaba-e82c-4298-b971-7818c5d739c8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:17.795996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:17.818793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1365/2712] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:17 2022] POST /identity/v3/projects => generated 405 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:17.822975 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69912bb5-94e2-4dfb-b3bb-4f0402b9aa05 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:17.833767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69912bb5-94e2-4dfb-b3bb-4f0402b9aa05 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:17.888343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6266e4b0-b90d-49f3-9665-a4dc8b73faff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:17.889002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6266e4b0-b90d-49f3-9665-a4dc8b73faff admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:17.889285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6266e4b0-b90d-49f3-9665-a4dc8b73faff admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:17.889558 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6266e4b0-b90d-49f3-9665-a4dc8b73faff admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:17.890211 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6266e4b0-b90d-49f3-9665-a4dc8b73faff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:17.892137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6266e4b0-b90d-49f3-9665-a4dc8b73faff admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:17.913154 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1348/2713] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:17 2022] POST /identity/v3/users => generated 303 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:17.917277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4336aaba-e82c-4298-b971-7818c5d739c8 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:17.927789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4336aaba-e82c-4298-b971-7818c5d739c8 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:17.932350 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c86fdd-77e0-40e6-92dd-9330a5079028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:17.932995 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c86fdd-77e0-40e6-92dd-9330a5079028 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:17.933282 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c86fdd-77e0-40e6-92dd-9330a5079028 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:17.933561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c86fdd-77e0-40e6-92dd-9330a5079028 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:17.934255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c86fdd-77e0-40e6-92dd-9330a5079028 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:17.935837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c86fdd-77e0-40e6-92dd-9330a5079028 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:17.945573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1366/2714] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:17.949943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6266e4b0-b90d-49f3-9665-a4dc8b73faff admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:17.961524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6266e4b0-b90d-49f3-9665-a4dc8b73faff admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:17.967090 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403d048e-4d8c-46e0-b132-515a89ca3b75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:17.967711 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403d048e-4d8c-46e0-b132-515a89ca3b75 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:17.968001 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403d048e-4d8c-46e0-b132-515a89ca3b75 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:17.968284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403d048e-4d8c-46e0-b132-515a89ca3b75 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:17.969038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403d048e-4d8c-46e0-b132-515a89ca3b75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:17.970639 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403d048e-4d8c-46e0-b132-515a89ca3b75 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:17.982175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1349/2715] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:17 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:17.986281 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c86fdd-77e0-40e6-92dd-9330a5079028 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:17.997043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c86fdd-77e0-40e6-92dd-9330a5079028 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:18.002252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e76d4c2-b1e7-4a7a-8d6b-5bcd966598e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:18.003117 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e76d4c2-b1e7-4a7a-8d6b-5bcd966598e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.003500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e76d4c2-b1e7-4a7a-8d6b-5bcd966598e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.003890 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e76d4c2-b1e7-4a7a-8d6b-5bcd966598e8 admin admin] PATH_INFO: `/v3/projects/b688ac47cf504748bed3fb1dbba913ce/users/e91c9fc61f314609b4349a0ab082d7c9/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.020166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e76d4c2-b1e7-4a7a-8d6b-5bcd966598e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b688ac47cf504748bed3fb1dbba913ce, user_id=e91c9fc61f314609b4349a0ab082d7c9, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:18.022816 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e76d4c2-b1e7-4a7a-8d6b-5bcd966598e8 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:18.037153 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1367/2716] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:17 2022] PUT /identity/v3/projects/b688ac47cf504748bed3fb1dbba913ce/users/e91c9fc61f314609b4349a0ab082d7c9/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:18.048759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8838f555-a29d-43f7-b5cf-86bb403f1aec None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.049079 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8838f555-a29d-43f7-b5cf-86bb403f1aec None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.049369 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8838f555-a29d-43f7-b5cf-86bb403f1aec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.079392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-8838f555-a29d-43f7-b5cf-86bb403f1aec None None] MFA Rules not processed for user `e91c9fc61f314609b4349a0ab082d7c9`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:18.080649 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8838f555-a29d-43f7-b5cf-86bb403f1aec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:18.148255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1350/2717] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:18 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:18.152722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e76d4c2-b1e7-4a7a-8d6b-5bcd966598e8 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:18.213671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e76d4c2-b1e7-4a7a-8d6b-5bcd966598e8 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:18.218643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35030844-c19c-41dc-ad60-53510e288808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:18.219255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35030844-c19c-41dc-ad60-53510e288808 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.219539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35030844-c19c-41dc-ad60-53510e288808 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.219841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35030844-c19c-41dc-ad60-53510e288808 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.220844 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35030844-c19c-41dc-ad60-53510e288808 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:18.222725 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35030844-c19c-41dc-ad60-53510e288808 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:18.249607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1368/2718] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:18 2022] POST /identity/v3/projects => generated 405 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:18.253814 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8838f555-a29d-43f7-b5cf-86bb403f1aec None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:18.265007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8838f555-a29d-43f7-b5cf-86bb403f1aec None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:18.282804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:18.321541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a348e38b-c2d9-49eb-b767-0f7e687615be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:18.322156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a348e38b-c2d9-49eb-b767-0f7e687615be admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.322439 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a348e38b-c2d9-49eb-b767-0f7e687615be admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.322714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a348e38b-c2d9-49eb-b767-0f7e687615be admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.323375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a348e38b-c2d9-49eb-b767-0f7e687615be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:18.325919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a348e38b-c2d9-49eb-b767-0f7e687615be admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:18.347421 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1351/2719] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:18 2022] POST /identity/v3/users => generated 309 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:18.351320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35030844-c19c-41dc-ad60-53510e288808 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:18.362000 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35030844-c19c-41dc-ad60-53510e288808 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:18.366570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e682b7-a4ef-4fb8-ab42-4fcefa630a09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:18.367173 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e682b7-a4ef-4fb8-ab42-4fcefa630a09 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.367456 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e682b7-a4ef-4fb8-ab42-4fcefa630a09 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.367728 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e682b7-a4ef-4fb8-ab42-4fcefa630a09 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.368411 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e682b7-a4ef-4fb8-ab42-4fcefa630a09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:18.370024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e682b7-a4ef-4fb8-ab42-4fcefa630a09 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:18.380578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1369/2720] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:18.384377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a348e38b-c2d9-49eb-b767-0f7e687615be admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:18.394384 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a348e38b-c2d9-49eb-b767-0f7e687615be admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:18.399076 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d74298-44f5-48d5-9945-9265c9f9b00b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:18.399737 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d74298-44f5-48d5-9945-9265c9f9b00b admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.400021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d74298-44f5-48d5-9945-9265c9f9b00b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.400300 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d74298-44f5-48d5-9945-9265c9f9b00b admin admin] PATH_INFO: `/v3/projects/3470e637ae0c4a8fb9a141c5d8d4bc6e/users/6f377da15e044316b5ef227253d224ab/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.415842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d74298-44f5-48d5-9945-9265c9f9b00b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3470e637ae0c4a8fb9a141c5d8d4bc6e, user_id=6f377da15e044316b5ef227253d224ab, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:18.419648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d74298-44f5-48d5-9945-9265c9f9b00b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:18.434096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1352/2721] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:18 2022] PUT /identity/v3/projects/3470e637ae0c4a8fb9a141c5d8d4bc6e/users/6f377da15e044316b5ef227253d224ab/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:18.438239 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e682b7-a4ef-4fb8-ab42-4fcefa630a09 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:18.449858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e682b7-a4ef-4fb8-ab42-4fcefa630a09 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:18.505671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f27c6e-b280-48b3-bc18-362e5d6d2f45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:18.506293 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f27c6e-b280-48b3-bc18-362e5d6d2f45 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.506585 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f27c6e-b280-48b3-bc18-362e5d6d2f45 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.506868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f27c6e-b280-48b3-bc18-362e5d6d2f45 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.507565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f27c6e-b280-48b3-bc18-362e5d6d2f45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:18.509193 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f27c6e-b280-48b3-bc18-362e5d6d2f45 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:18.520107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1370/2722] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:18 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:18.524377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d74298-44f5-48d5-9945-9265c9f9b00b admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:18.535286 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d74298-44f5-48d5-9945-9265c9f9b00b admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:18.539892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbb916a-ad93-47d3-9b75-1117c8a7f528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:18.540546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbb916a-ad93-47d3-9b75-1117c8a7f528 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.540867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbb916a-ad93-47d3-9b75-1117c8a7f528 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.541173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbb916a-ad93-47d3-9b75-1117c8a7f528 admin admin] PATH_INFO: `/v3/projects/3470e637ae0c4a8fb9a141c5d8d4bc6e/users/6f377da15e044316b5ef227253d224ab/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.543134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbb916a-ad93-47d3-9b75-1117c8a7f528 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3470e637ae0c4a8fb9a141c5d8d4bc6e, user_id=6f377da15e044316b5ef227253d224ab, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:18.546068 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbb916a-ad93-47d3-9b75-1117c8a7f528 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:18.556140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1353/2723] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:18 2022] PUT /identity/v3/projects/3470e637ae0c4a8fb9a141c5d8d4bc6e/users/6f377da15e044316b5ef227253d224ab/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:18.566289 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72180fe1-c39e-4ff0-8cf8-27e129970f35 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.566588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72180fe1-c39e-4ff0-8cf8-27e129970f35 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.566874 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72180fe1-c39e-4ff0-8cf8-27e129970f35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.599571 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-72180fe1-c39e-4ff0-8cf8-27e129970f35 None None] MFA Rules not processed for user `6f377da15e044316b5ef227253d224ab`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:18.600820 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-72180fe1-c39e-4ff0-8cf8-27e129970f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:18.609949 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:18.670190 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1371/2724] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:18 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:18.680712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbb916a-ad93-47d3-9b75-1117c8a7f528 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:18.682397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-dbbb916a-ad93-47d3-9b75-1117c8a7f528 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:18.738871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbb916a-ad93-47d3-9b75-1117c8a7f528 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:18.742503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bd8ca0-1aca-4f40-bd2c-65298c3a19da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:18.743109 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bd8ca0-1aca-4f40-bd2c-65298c3a19da service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.743393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bd8ca0-1aca-4f40-bd2c-65298c3a19da service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.743669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bd8ca0-1aca-4f40-bd2c-65298c3a19da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.753304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bd8ca0-1aca-4f40-bd2c-65298c3a19da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:18.754717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bd8ca0-1aca-4f40-bd2c-65298c3a19da service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:18.764726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1354/2725] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:18 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:18.921716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72180fe1-c39e-4ff0-8cf8-27e129970f35 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:18.984003 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72180fe1-c39e-4ff0-8cf8-27e129970f35 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:18.988720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:18.989372 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:18.989666 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:18.989952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] PATH_INFO: `/v3/users/e91c9fc61f314609b4349a0ab082d7c9` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:18.991151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e91c9fc61f314609b4349a0ab082d7c9)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:18.993133 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:19.046176 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e91c9fc61f314609b4349a0ab082d7c9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.046842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e91c9fc61f314609b4349a0ab082d7c9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.047558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e91c9fc61f314609b4349a0ab082d7c9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.063855 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e91c9fc61f314609b4349a0ab082d7c9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.070624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e91c9fc61f314609b4349a0ab082d7c9'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.084588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1372/2726] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:18 2022] DELETE /identity/v3/users/e91c9fc61f314609b4349a0ab082d7c9 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:19.089131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6458902-04da-4900-aa32-4cec979023ce None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:19.089732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6458902-04da-4900-aa32-4cec979023ce None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:19.089732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6458902-04da-4900-aa32-4cec979023ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:19.108702 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-d6458902-04da-4900-aa32-4cec979023ce None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:19.109910 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d6458902-04da-4900-aa32-4cec979023ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:19.167404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1355/2727] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:19.178235 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:19.179648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:19.211189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:19.240784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b490aa0-1926-4f84-bab2-ae4816d00a5e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:19.244307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81269c92-4f9f-4b1c-bf07-44e5f25ac47c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:19.244951 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81269c92-4f9f-4b1c-bf07-44e5f25ac47c service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:19.245231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81269c92-4f9f-4b1c-bf07-44e5f25ac47c service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:19.245507 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81269c92-4f9f-4b1c-bf07-44e5f25ac47c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:19.254821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81269c92-4f9f-4b1c-bf07-44e5f25ac47c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:19.256221 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81269c92-4f9f-4b1c-bf07-44e5f25ac47c service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:19.265327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1373/2728] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:19.529199 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6458902-04da-4900-aa32-4cec979023ce None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:19.530915 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d6458902-04da-4900-aa32-4cec979023ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:19.545267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6458902-04da-4900-aa32-4cec979023ce None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:19.549774 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:19.550418 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:19.550700 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:19.550980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] PATH_INFO: `/v3/projects/b688ac47cf504748bed3fb1dbba913ce` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:19.552862 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b688ac47cf504748bed3fb1dbba913ce)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:19.554759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:19.566623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:19.602364 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b688ac47cf504748bed3fb1dbba913ce', 'request_id': 'req-c8697a16-d752-4a59-989e-0742de03503e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.610046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b688ac47cf504748bed3fb1dbba913ce', 'request_id': 'req-c8697a16-d752-4a59-989e-0742de03503e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.616155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b688ac47cf504748bed3fb1dbba913ce', 'request_id': 'req-c8697a16-d752-4a59-989e-0742de03503e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.627134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] The token cache is being invalidate because project b688ac47cf504748bed3fb1dbba913ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:19.627436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.628116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.629664 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1356/2729] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:19 2022] DELETE /identity/v3/projects/b688ac47cf504748bed3fb1dbba913ce => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:19.633668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81269c92-4f9f-4b1c-bf07-44e5f25ac47c service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:19.643288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81269c92-4f9f-4b1c-bf07-44e5f25ac47c service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:19.700455 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:19.701121 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:19.701410 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:19.701691 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] PATH_INFO: `/v3/users/6f377da15e044316b5ef227253d224ab` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:19.702909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f377da15e044316b5ef227253d224ab)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:19.704799 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:19.748765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f377da15e044316b5ef227253d224ab'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.749407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f377da15e044316b5ef227253d224ab'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.750021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f377da15e044316b5ef227253d224ab'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.762114 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f377da15e044316b5ef227253d224ab'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.768897 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-040260e3-26a2-4525-b664-338e3f8c1b31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f377da15e044316b5ef227253d224ab'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:19.782589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1374/2730] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:19 2022] DELETE /identity/v3/users/6f377da15e044316b5ef227253d224ab => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:20.052961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.054389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:20.083215 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8697a16-d752-4a59-989e-0742de03503e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:20.141128 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:20.141805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.142057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.142340 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] PATH_INFO: `/v3/projects/3470e637ae0c4a8fb9a141c5d8d4bc6e` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.143506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3470e637ae0c4a8fb9a141c5d8d4bc6e)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:20.145464 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:20.190691 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3470e637ae0c4a8fb9a141c5d8d4bc6e', 'request_id': 'req-c5828cbb-550d-469e-895b-e231de263b57'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:20.197115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3470e637ae0c4a8fb9a141c5d8d4bc6e', 'request_id': 'req-c5828cbb-550d-469e-895b-e231de263b57'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:20.203196 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3470e637ae0c4a8fb9a141c5d8d4bc6e', 'request_id': 'req-c5828cbb-550d-469e-895b-e231de263b57'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:20.214329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] The token cache is being invalidate because project 3470e637ae0c4a8fb9a141c5d8d4bc6e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:20.215386 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:20.216180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c5828cbb-550d-469e-895b-e231de263b57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:20.217709 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1357/2731] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:20 2022] DELETE /identity/v3/projects/3470e637ae0c4a8fb9a141c5d8d4bc6e => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:20.222626 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6959e7d-b973-4c6a-95f8-19e37afe16e5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.222918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6959e7d-b973-4c6a-95f8-19e37afe16e5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.223208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6959e7d-b973-4c6a-95f8-19e37afe16e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.242394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d6959e7d-b973-4c6a-95f8-19e37afe16e5 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:20.243608 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d6959e7d-b973-4c6a-95f8-19e37afe16e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:20.287090 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:20.303976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1375/2732] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:20.313714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e06298-4751-452d-b08f-9dec84b15dec None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.314005 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e06298-4751-452d-b08f-9dec84b15dec None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.314283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e06298-4751-452d-b08f-9dec84b15dec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.335584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c9e06298-4751-452d-b08f-9dec84b15dec None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:20.336826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c9e06298-4751-452d-b08f-9dec84b15dec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:20.344854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1358/2733] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:20.349130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6959e7d-b973-4c6a-95f8-19e37afe16e5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.358693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6959e7d-b973-4c6a-95f8-19e37afe16e5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:20.363212 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16e8263-7225-48c3-906c-76fcde4f97d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:20.363818 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16e8263-7225-48c3-906c-76fcde4f97d4 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.364102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16e8263-7225-48c3-906c-76fcde4f97d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.364379 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16e8263-7225-48c3-906c-76fcde4f97d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.365116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16e8263-7225-48c3-906c-76fcde4f97d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 05:00:20.365423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16e8263-7225-48c3-906c-76fcde4f97d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:20.367008 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16e8263-7225-48c3-906c-76fcde4f97d4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:20.378796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1376/2734] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 05:00:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 05:00:20.382996 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e06298-4751-452d-b08f-9dec84b15dec None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.393097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e06298-4751-452d-b08f-9dec84b15dec None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:20.397762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7549ba93-f231-4092-aeff-cfbb0a4e12e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:20.398368 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7549ba93-f231-4092-aeff-cfbb0a4e12e6 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.398659 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7549ba93-f231-4092-aeff-cfbb0a4e12e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.398939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7549ba93-f231-4092-aeff-cfbb0a4e12e6 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.399605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7549ba93-f231-4092-aeff-cfbb0a4e12e6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:20.401555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7549ba93-f231-4092-aeff-cfbb0a4e12e6 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:20.428162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1359/2735] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:20 2022] POST /identity/v3/projects => generated 387 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:20.432226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16e8263-7225-48c3-906c-76fcde4f97d4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.442733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16e8263-7225-48c3-906c-76fcde4f97d4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:20.498220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2f716d-82c9-451c-825a-880d73bcede7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:20.498762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2f716d-82c9-451c-825a-880d73bcede7 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.499050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2f716d-82c9-451c-825a-880d73bcede7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.499332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2f716d-82c9-451c-825a-880d73bcede7 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.500010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2f716d-82c9-451c-825a-880d73bcede7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:20.501945 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2f716d-82c9-451c-825a-880d73bcede7 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:20.523111 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1377/2736] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:20 2022] POST /identity/v3/users => generated 294 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:00:20.527890 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7549ba93-f231-4092-aeff-cfbb0a4e12e6 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.533275 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:20.539435 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7549ba93-f231-4092-aeff-cfbb0a4e12e6 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:20.546429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920d6971-52b2-44fc-97c0-57b2507a0f17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:20.547045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920d6971-52b2-44fc-97c0-57b2507a0f17 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.547328 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920d6971-52b2-44fc-97c0-57b2507a0f17 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.547605 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920d6971-52b2-44fc-97c0-57b2507a0f17 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.548297 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920d6971-52b2-44fc-97c0-57b2507a0f17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:20.549960 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920d6971-52b2-44fc-97c0-57b2507a0f17 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:20.560704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1360/2737] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:20 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:20.564542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2f716d-82c9-451c-825a-880d73bcede7 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.575458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2f716d-82c9-451c-825a-880d73bcede7 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:20.580041 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0db2cbc-4fb5-44bc-8359-0f672e896573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:20.580678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0db2cbc-4fb5-44bc-8359-0f672e896573 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.580993 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0db2cbc-4fb5-44bc-8359-0f672e896573 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.581269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0db2cbc-4fb5-44bc-8359-0f672e896573 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.581953 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0db2cbc-4fb5-44bc-8359-0f672e896573 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:20.583518 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0db2cbc-4fb5-44bc-8359-0f672e896573 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:20.594390 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1378/2738] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:20.598316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920d6971-52b2-44fc-97c0-57b2507a0f17 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.608594 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920d6971-52b2-44fc-97c0-57b2507a0f17 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:20.613235 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87c5b9d-3873-4516-8216-1cf354189a0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:20.613882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87c5b9d-3873-4516-8216-1cf354189a0a admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.614165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87c5b9d-3873-4516-8216-1cf354189a0a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.614461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87c5b9d-3873-4516-8216-1cf354189a0a admin admin] PATH_INFO: `/v3/projects/eb79e32a06794825948e3332ce675323/users/7c1be47c4d16485db918dc59766cd007/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.629167 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87c5b9d-3873-4516-8216-1cf354189a0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb79e32a06794825948e3332ce675323, user_id=7c1be47c4d16485db918dc59766cd007, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:20.631782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87c5b9d-3873-4516-8216-1cf354189a0a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:20.645492 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1361/2739] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:20 2022] PUT /identity/v3/projects/eb79e32a06794825948e3332ce675323/users/7c1be47c4d16485db918dc59766cd007/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:20.657641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2be643-ec1c-4706-8696-f46e3a705845 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.658096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2be643-ec1c-4706-8696-f46e3a705845 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.658193 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2be643-ec1c-4706-8696-f46e3a705845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.686438 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-6a2be643-ec1c-4706-8696-f46e3a705845 None None] MFA Rules not processed for user `7c1be47c4d16485db918dc59766cd007`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:20.687650 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-6a2be643-ec1c-4706-8696-f46e3a705845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:20.749360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1379/2740] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 05:00:20.753833 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87c5b9d-3873-4516-8216-1cf354189a0a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.814838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87c5b9d-3873-4516-8216-1cf354189a0a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:20.819545 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43106aba-a3e0-4b47-bafc-2e5a7842ab7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:20.820164 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43106aba-a3e0-4b47-bafc-2e5a7842ab7f admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.820454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43106aba-a3e0-4b47-bafc-2e5a7842ab7f admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.820776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43106aba-a3e0-4b47-bafc-2e5a7842ab7f admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.821452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43106aba-a3e0-4b47-bafc-2e5a7842ab7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:20.823343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43106aba-a3e0-4b47-bafc-2e5a7842ab7f admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:20.850533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1362/2741] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:20 2022] POST /identity/v3/projects => generated 387 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:20.854586 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2be643-ec1c-4706-8696-f46e3a705845 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.865139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2be643-ec1c-4706-8696-f46e3a705845 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:20.906978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:20.922938 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c971fb4d-8f7a-4ec1-8e3e-350ec8474dd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:20.923565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c971fb4d-8f7a-4ec1-8e3e-350ec8474dd4 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.923856 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c971fb4d-8f7a-4ec1-8e3e-350ec8474dd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.924144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c971fb4d-8f7a-4ec1-8e3e-350ec8474dd4 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.924848 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c971fb4d-8f7a-4ec1-8e3e-350ec8474dd4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:20.926766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c971fb4d-8f7a-4ec1-8e3e-350ec8474dd4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:20.947696 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1380/2742] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:20 2022] POST /identity/v3/users => generated 300 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:20.952326 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43106aba-a3e0-4b47-bafc-2e5a7842ab7f admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.962587 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43106aba-a3e0-4b47-bafc-2e5a7842ab7f admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:20.967758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f57051-27b4-4c8c-b25c-b20816c81842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:20.968375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f57051-27b4-4c8c-b25c-b20816c81842 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:20.968717 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f57051-27b4-4c8c-b25c-b20816c81842 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:20.969024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f57051-27b4-4c8c-b25c-b20816c81842 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:20.969719 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f57051-27b4-4c8c-b25c-b20816c81842 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:20.971316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f57051-27b4-4c8c-b25c-b20816c81842 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:20.981670 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1363/2743] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:20.985640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c971fb4d-8f7a-4ec1-8e3e-350ec8474dd4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:20.995515 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c971fb4d-8f7a-4ec1-8e3e-350ec8474dd4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:21.000911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec355b-8482-4f1c-977d-0c8533d25460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:21.001509 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec355b-8482-4f1c-977d-0c8533d25460 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:21.001795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec355b-8482-4f1c-977d-0c8533d25460 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:21.002076 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec355b-8482-4f1c-977d-0c8533d25460 admin admin] PATH_INFO: `/v3/projects/5dc6ae190d7c49c980def03e7c4ef9d2/users/40186d30a16842fd9765333a72b15350/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:21.017223 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec355b-8482-4f1c-977d-0c8533d25460 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5dc6ae190d7c49c980def03e7c4ef9d2, user_id=40186d30a16842fd9765333a72b15350, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:21.020057 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec355b-8482-4f1c-977d-0c8533d25460 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:21.033115 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1381/2744] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:20 2022] PUT /identity/v3/projects/5dc6ae190d7c49c980def03e7c4ef9d2/users/40186d30a16842fd9765333a72b15350/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:21.037021 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f57051-27b4-4c8c-b25c-b20816c81842 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:21.047456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f57051-27b4-4c8c-b25c-b20816c81842 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:21.102938 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322e6be3-8a25-4d0b-a6c3-eac776d33e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:21.103590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e6be3-8a25-4d0b-a6c3-eac776d33e71 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:21.103904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e6be3-8a25-4d0b-a6c3-eac776d33e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:21.104209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e6be3-8a25-4d0b-a6c3-eac776d33e71 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:21.104941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322e6be3-8a25-4d0b-a6c3-eac776d33e71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:21.106830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322e6be3-8a25-4d0b-a6c3-eac776d33e71 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:21.119893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1364/2745] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:21 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:21.121976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec355b-8482-4f1c-977d-0c8533d25460 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:21.132535 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec355b-8482-4f1c-977d-0c8533d25460 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:21.137750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e3a07a-2aa4-4589-aead-ee3cc1f08e8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:21.138415 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e3a07a-2aa4-4589-aead-ee3cc1f08e8d admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:21.138865 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e3a07a-2aa4-4589-aead-ee3cc1f08e8d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:21.138958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e3a07a-2aa4-4589-aead-ee3cc1f08e8d admin admin] PATH_INFO: `/v3/projects/5dc6ae190d7c49c980def03e7c4ef9d2/users/40186d30a16842fd9765333a72b15350/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:21.140988 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e3a07a-2aa4-4589-aead-ee3cc1f08e8d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5dc6ae190d7c49c980def03e7c4ef9d2, user_id=40186d30a16842fd9765333a72b15350, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:21.143590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8e3a07a-2aa4-4589-aead-ee3cc1f08e8d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:21.154431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1382/2746] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:21 2022] PUT /identity/v3/projects/5dc6ae190d7c49c980def03e7c4ef9d2/users/40186d30a16842fd9765333a72b15350/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:21.164685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f2188-37ac-45fc-956b-34ae7fe7177e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:21.165188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f2188-37ac-45fc-956b-34ae7fe7177e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:21.165387 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f2188-37ac-45fc-956b-34ae7fe7177e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:21.193044 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-d25f2188-37ac-45fc-956b-34ae7fe7177e None None] MFA Rules not processed for user `40186d30a16842fd9765333a72b15350`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:21.194248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d25f2188-37ac-45fc-956b-34ae7fe7177e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:21.219831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:21.262777 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1365/2747] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:21 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:21.273752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e3a07a-2aa4-4589-aead-ee3cc1f08e8d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:21.275465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f8e3a07a-2aa4-4589-aead-ee3cc1f08e8d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:21.333578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e3a07a-2aa4-4589-aead-ee3cc1f08e8d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:21.337710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a04983-9b93-4429-a95f-bb4d56866ade service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:21.339135 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a04983-9b93-4429-a95f-bb4d56866ade service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:21.339471 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a04983-9b93-4429-a95f-bb4d56866ade service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:21.339749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a04983-9b93-4429-a95f-bb4d56866ade service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:21.348898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a04983-9b93-4429-a95f-bb4d56866ade service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:21.350347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a04983-9b93-4429-a95f-bb4d56866ade service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:21.360151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1383/2748] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:21 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:21.389696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25f2188-37ac-45fc-956b-34ae7fe7177e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:21.458623 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25f2188-37ac-45fc-956b-34ae7fe7177e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:21.463136 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:21.463771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:21.464052 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:21.464391 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] PATH_INFO: `/v3/users/7c1be47c4d16485db918dc59766cd007` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:21.465592 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c1be47c4d16485db918dc59766cd007)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:21.467493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:21.512012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c1be47c4d16485db918dc59766cd007'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:21.519221 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c1be47c4d16485db918dc59766cd007'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:21.519762 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c1be47c4d16485db918dc59766cd007'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:21.520358 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c1be47c4d16485db918dc59766cd007'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:21.527287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c1be47c4d16485db918dc59766cd007'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:21.545116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1366/2749] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:21 2022] DELETE /identity/v3/users/7c1be47c4d16485db918dc59766cd007 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:21.549591 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836d92e0-0b39-4e2d-b539-05d97e560a83 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:21.549886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836d92e0-0b39-4e2d-b539-05d97e560a83 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:21.550169 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836d92e0-0b39-4e2d-b539-05d97e560a83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:21.570710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-836d92e0-0b39-4e2d-b539-05d97e560a83 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:21.571947 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-836d92e0-0b39-4e2d-b539-05d97e560a83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:21.630726 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1384/2750] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:21.640265 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:21.641787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:21.689667 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:21.703097 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946416e1-2e50-44b0-a709-10ab208a9d63 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:21.706875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa42bd61-9270-4d65-b163-606b3d0d6934 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:21.707494 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa42bd61-9270-4d65-b163-606b3d0d6934 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:21.707788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa42bd61-9270-4d65-b163-606b3d0d6934 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:21.708069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa42bd61-9270-4d65-b163-606b3d0d6934 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:21.719461 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa42bd61-9270-4d65-b163-606b3d0d6934 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:21.720934 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa42bd61-9270-4d65-b163-606b3d0d6934 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:21.732788 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1367/2751] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:22.004805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836d92e0-0b39-4e2d-b539-05d97e560a83 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:22.006279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-836d92e0-0b39-4e2d-b539-05d97e560a83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:22.019186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836d92e0-0b39-4e2d-b539-05d97e560a83 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:22.023746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:22.024391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:22.024714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:22.024997 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] PATH_INFO: `/v3/projects/eb79e32a06794825948e3332ce675323` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:22.026145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb79e32a06794825948e3332ce675323)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:22.029031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:22.067783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:22.079732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb79e32a06794825948e3332ce675323', 'request_id': 'req-538e1716-173b-4ca4-821b-e4cc083a390f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.085762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb79e32a06794825948e3332ce675323', 'request_id': 'req-538e1716-173b-4ca4-821b-e4cc083a390f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.093748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb79e32a06794825948e3332ce675323', 'request_id': 'req-538e1716-173b-4ca4-821b-e4cc083a390f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.104348 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] The token cache is being invalidate because project eb79e32a06794825948e3332ce675323 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:22.104672 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.105629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.107005 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1385/2752] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:22 2022] DELETE /identity/v3/projects/eb79e32a06794825948e3332ce675323 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:22.111029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa42bd61-9270-4d65-b163-606b3d0d6934 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:22.121147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa42bd61-9270-4d65-b163-606b3d0d6934 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:22.177410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:22.178055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:22.178343 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:22.178626 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] PATH_INFO: `/v3/users/40186d30a16842fd9765333a72b15350` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:22.179905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] RBAC: Authorizing `identity:delete_user(user_id=40186d30a16842fd9765333a72b15350)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:22.181844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:22.231170 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40186d30a16842fd9765333a72b15350'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.237725 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40186d30a16842fd9765333a72b15350'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.238329 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40186d30a16842fd9765333a72b15350'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.238947 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40186d30a16842fd9765333a72b15350'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.245723 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-75a9ea99-9163-4a9b-86bf-11cd81c98506 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40186d30a16842fd9765333a72b15350'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.263171 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1368/2753] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:22 2022] DELETE /identity/v3/users/40186d30a16842fd9765333a72b15350 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:22.538353 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:22.539872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:22.553573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538e1716-173b-4ca4-821b-e4cc083a390f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:22.610937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:22.611607 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:22.611899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:22.612225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] PATH_INFO: `/v3/projects/5dc6ae190d7c49c980def03e7c4ef9d2` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:22.613440 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5dc6ae190d7c49c980def03e7c4ef9d2)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:22.615391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:22.660708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5dc6ae190d7c49c980def03e7c4ef9d2', 'request_id': 'req-3804ca05-eebf-41f4-8617-af28bffab74f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.666939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5dc6ae190d7c49c980def03e7c4ef9d2', 'request_id': 'req-3804ca05-eebf-41f4-8617-af28bffab74f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.674757 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5dc6ae190d7c49c980def03e7c4ef9d2', 'request_id': 'req-3804ca05-eebf-41f4-8617-af28bffab74f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.686167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] The token cache is being invalidate because project 5dc6ae190d7c49c980def03e7c4ef9d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:22.686541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.687172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-3804ca05-eebf-41f4-8617-af28bffab74f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:22.688788 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1386/2754] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:22 2022] DELETE /identity/v3/projects/5dc6ae190d7c49c980def03e7c4ef9d2 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:22.693690 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4412b043-ddbc-4844-b3fd-7d041a7b5ff7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:22.693981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4412b043-ddbc-4844-b3fd-7d041a7b5ff7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:22.694264 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4412b043-ddbc-4844-b3fd-7d041a7b5ff7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:22.714596 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-4412b043-ddbc-4844-b3fd-7d041a7b5ff7 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:22.715918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4412b043-ddbc-4844-b3fd-7d041a7b5ff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:22.774224 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1369/2755] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:22.838048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f076f78e-6a2f-49b2-b52f-4cec29a41c3f None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:22.838266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f076f78e-6a2f-49b2-b52f-4cec29a41c3f None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:22.838516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f076f78e-6a2f-49b2-b52f-4cec29a41c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:22.858383 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-f076f78e-6a2f-49b2-b52f-4cec29a41c3f None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:22.859603 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f076f78e-6a2f-49b2-b52f-4cec29a41c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:22.867472 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1387/2756] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:22.872553 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4412b043-ddbc-4844-b3fd-7d041a7b5ff7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:22.878645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:22.882404 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4412b043-ddbc-4844-b3fd-7d041a7b5ff7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:22.888604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6654a80-8db0-4267-b30b-6c4db76592f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:22.889429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6654a80-8db0-4267-b30b-6c4db76592f3 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:22.889758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6654a80-8db0-4267-b30b-6c4db76592f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:22.890178 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6654a80-8db0-4267-b30b-6c4db76592f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:22.890940 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6654a80-8db0-4267-b30b-6c4db76592f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 05:00:22.891410 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6654a80-8db0-4267-b30b-6c4db76592f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:22.893062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6654a80-8db0-4267-b30b-6c4db76592f3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:22.904688 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1370/2757] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 05:00:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 05:00:22.908982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f076f78e-6a2f-49b2-b52f-4cec29a41c3f None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:22.918937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f076f78e-6a2f-49b2-b52f-4cec29a41c3f None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:22.923656 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fdb88a-d4d3-46f7-98d3-ca4ad32846ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:22.924260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fdb88a-d4d3-46f7-98d3-ca4ad32846ed admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:22.924553 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fdb88a-d4d3-46f7-98d3-ca4ad32846ed admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:22.924863 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fdb88a-d4d3-46f7-98d3-ca4ad32846ed admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:22.925533 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fdb88a-d4d3-46f7-98d3-ca4ad32846ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:22.927425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36fdb88a-d4d3-46f7-98d3-ca4ad32846ed admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:22.957520 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1388/2758] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:22 2022] POST /identity/v3/projects => generated 387 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:22.958598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6654a80-8db0-4267-b30b-6c4db76592f3 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:22.968437 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6654a80-8db0-4267-b30b-6c4db76592f3 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.022523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4203d7-b5f2-4d38-9d47-d4c842ace34c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.023133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4203d7-b5f2-4d38-9d47-d4c842ace34c admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.023412 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4203d7-b5f2-4d38-9d47-d4c842ace34c admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.023693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4203d7-b5f2-4d38-9d47-d4c842ace34c admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.024507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4203d7-b5f2-4d38-9d47-d4c842ace34c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.026449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4203d7-b5f2-4d38-9d47-d4c842ace34c admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.047216 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1371/2759] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:22 2022] POST /identity/v3/users => generated 294 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:00:23.051336 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fdb88a-d4d3-46f7-98d3-ca4ad32846ed admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.062175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fdb88a-d4d3-46f7-98d3-ca4ad32846ed admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.066777 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70a147f-61c2-4f20-8620-67cd3860c7df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.067387 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a147f-61c2-4f20-8620-67cd3860c7df admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.067685 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a147f-61c2-4f20-8620-67cd3860c7df admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.067952 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a147f-61c2-4f20-8620-67cd3860c7df admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.068671 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70a147f-61c2-4f20-8620-67cd3860c7df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.070254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70a147f-61c2-4f20-8620-67cd3860c7df admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.080045 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1389/2760] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:23 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:23.083874 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4203d7-b5f2-4d38-9d47-d4c842ace34c admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.093974 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4203d7-b5f2-4d38-9d47-d4c842ace34c admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.098544 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66452c2-975e-4c7e-a966-7cc8f474bda8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.099147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66452c2-975e-4c7e-a966-7cc8f474bda8 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.099430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66452c2-975e-4c7e-a966-7cc8f474bda8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.099708 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66452c2-975e-4c7e-a966-7cc8f474bda8 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.100397 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66452c2-975e-4c7e-a966-7cc8f474bda8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.102067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66452c2-975e-4c7e-a966-7cc8f474bda8 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.113058 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1372/2761] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:23.117014 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70a147f-61c2-4f20-8620-67cd3860c7df admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.127559 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70a147f-61c2-4f20-8620-67cd3860c7df admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.132271 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0318ae-1b21-4b5e-8eb4-a2656103d998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.133083 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0318ae-1b21-4b5e-8eb4-a2656103d998 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.133414 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0318ae-1b21-4b5e-8eb4-a2656103d998 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.133728 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0318ae-1b21-4b5e-8eb4-a2656103d998 admin admin] PATH_INFO: `/v3/projects/e8e3f3b6b7ea4c5db3e73497c47fed2b/users/77ee05a1b35d4fafa7ece47c44fa933f/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.145008 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:23.149906 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0318ae-1b21-4b5e-8eb4-a2656103d998 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8e3f3b6b7ea4c5db3e73497c47fed2b, user_id=77ee05a1b35d4fafa7ece47c44fa933f, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.153522 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0318ae-1b21-4b5e-8eb4-a2656103d998 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.166386 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1390/2762] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:23 2022] PUT /identity/v3/projects/e8e3f3b6b7ea4c5db3e73497c47fed2b/users/77ee05a1b35d4fafa7ece47c44fa933f/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:23.177331 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c2188d-1921-497d-9e2f-af36d814c582 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.177620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c2188d-1921-497d-9e2f-af36d814c582 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.178152 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c2188d-1921-497d-9e2f-af36d814c582 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.205577 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-25c2188d-1921-497d-9e2f-af36d814c582 None None] MFA Rules not processed for user `77ee05a1b35d4fafa7ece47c44fa933f`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:23.206828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-25c2188d-1921-497d-9e2f-af36d814c582 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:23.275078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1373/2763] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:23 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:23.280101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0318ae-1b21-4b5e-8eb4-a2656103d998 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.342917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0318ae-1b21-4b5e-8eb4-a2656103d998 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.347664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0debe37-9702-4e12-8f9f-6007dd5d9300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.348196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0debe37-9702-4e12-8f9f-6007dd5d9300 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.348479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0debe37-9702-4e12-8f9f-6007dd5d9300 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.348787 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0debe37-9702-4e12-8f9f-6007dd5d9300 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.349443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0debe37-9702-4e12-8f9f-6007dd5d9300 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.351318 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0debe37-9702-4e12-8f9f-6007dd5d9300 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.380204 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1391/2764] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:23 2022] POST /identity/v3/projects => generated 385 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:00:23.384260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c2188d-1921-497d-9e2f-af36d814c582 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.395593 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c2188d-1921-497d-9e2f-af36d814c582 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.452754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d872732b-4dde-4e8c-8ccd-cadac896cf92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.453377 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d872732b-4dde-4e8c-8ccd-cadac896cf92 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.453661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d872732b-4dde-4e8c-8ccd-cadac896cf92 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.453941 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d872732b-4dde-4e8c-8ccd-cadac896cf92 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.454656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d872732b-4dde-4e8c-8ccd-cadac896cf92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.456635 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d872732b-4dde-4e8c-8ccd-cadac896cf92 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.464981 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:23.478918 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1374/2765] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:23 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:23.484151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0debe37-9702-4e12-8f9f-6007dd5d9300 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.494334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0debe37-9702-4e12-8f9f-6007dd5d9300 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.498867 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3212862-6a8c-403f-9050-de43917918c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.499469 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3212862-6a8c-403f-9050-de43917918c4 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.499750 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3212862-6a8c-403f-9050-de43917918c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.500023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3212862-6a8c-403f-9050-de43917918c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.500737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3212862-6a8c-403f-9050-de43917918c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.502302 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3212862-6a8c-403f-9050-de43917918c4 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.512864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1392/2766] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:23 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:23.516942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d872732b-4dde-4e8c-8ccd-cadac896cf92 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.527849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d872732b-4dde-4e8c-8ccd-cadac896cf92 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.532698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870f633f-2db1-457d-ac76-593503f8f027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.533316 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f633f-2db1-457d-ac76-593503f8f027 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.533618 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f633f-2db1-457d-ac76-593503f8f027 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.533881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870f633f-2db1-457d-ac76-593503f8f027 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.534572 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870f633f-2db1-457d-ac76-593503f8f027 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.536162 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870f633f-2db1-457d-ac76-593503f8f027 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.547125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1375/2767] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:23.551186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3212862-6a8c-403f-9050-de43917918c4 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.561343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3212862-6a8c-403f-9050-de43917918c4 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.566165 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c829cf23-c1bd-4fcf-bbfd-567667a50ae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.566852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c829cf23-c1bd-4fcf-bbfd-567667a50ae6 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.567181 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c829cf23-c1bd-4fcf-bbfd-567667a50ae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.567464 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c829cf23-c1bd-4fcf-bbfd-567667a50ae6 admin admin] PATH_INFO: `/v3/projects/2e38be77337b47979eaf910db1cf0c59/users/ba2a01ae9ffa471b9f7001091dc4120e/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.584260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c829cf23-c1bd-4fcf-bbfd-567667a50ae6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e38be77337b47979eaf910db1cf0c59, user_id=ba2a01ae9ffa471b9f7001091dc4120e, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.587015 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c829cf23-c1bd-4fcf-bbfd-567667a50ae6 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.600366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1393/2768] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:23 2022] PUT /identity/v3/projects/2e38be77337b47979eaf910db1cf0c59/users/ba2a01ae9ffa471b9f7001091dc4120e/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:23.611026 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce29b38-4fb1-4886-9f24-3c17819d21e7 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.611322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce29b38-4fb1-4886-9f24-3c17819d21e7 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.611607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce29b38-4fb1-4886-9f24-3c17819d21e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.640887 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-cce29b38-4fb1-4886-9f24-3c17819d21e7 None None] MFA Rules not processed for user `ba2a01ae9ffa471b9f7001091dc4120e`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:23.642277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-cce29b38-4fb1-4886-9f24-3c17819d21e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:23.711825 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1376/2769] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:23 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:23.717056 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c829cf23-c1bd-4fcf-bbfd-567667a50ae6 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.778296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c829cf23-c1bd-4fcf-bbfd-567667a50ae6 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.782939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d375e6-c465-4002-af36-9b26e90193ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.783549 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d375e6-c465-4002-af36-9b26e90193ee admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.783834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d375e6-c465-4002-af36-9b26e90193ee admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.784109 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d375e6-c465-4002-af36-9b26e90193ee admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.784804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d375e6-c465-4002-af36-9b26e90193ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.786683 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d375e6-c465-4002-af36-9b26e90193ee admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.812804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1394/2770] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:23 2022] POST /identity/v3/projects => generated 387 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:00:23.817041 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce29b38-4fb1-4886-9f24-3c17819d21e7 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.829336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce29b38-4fb1-4886-9f24-3c17819d21e7 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.895262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802c0ce-2022-49e2-899d-de32d64de225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.895996 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802c0ce-2022-49e2-899d-de32d64de225 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.896360 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802c0ce-2022-49e2-899d-de32d64de225 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.896740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802c0ce-2022-49e2-899d-de32d64de225 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.897500 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802c0ce-2022-49e2-899d-de32d64de225 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.899507 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802c0ce-2022-49e2-899d-de32d64de225 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.921181 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1377/2771] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:23 2022] POST /identity/v3/users => generated 300 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:00:23.925180 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d375e6-c465-4002-af36-9b26e90193ee admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.936229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d375e6-c465-4002-af36-9b26e90193ee admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.940975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba9f87e-aeab-41a0-bda8-419619ef5278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.941595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba9f87e-aeab-41a0-bda8-419619ef5278 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.941882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba9f87e-aeab-41a0-bda8-419619ef5278 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.942160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba9f87e-aeab-41a0-bda8-419619ef5278 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.942975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba9f87e-aeab-41a0-bda8-419619ef5278 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.944570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba9f87e-aeab-41a0-bda8-419619ef5278 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:23.956936 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1395/2772] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:23.961212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802c0ce-2022-49e2-899d-de32d64de225 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:23.972628 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802c0ce-2022-49e2-899d-de32d64de225 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:23.977251 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f8eb7d-32ff-4dc8-bf2f-54fa47f0707a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:23.977919 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f8eb7d-32ff-4dc8-bf2f-54fa47f0707a admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:23.978202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f8eb7d-32ff-4dc8-bf2f-54fa47f0707a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:23.978477 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f8eb7d-32ff-4dc8-bf2f-54fa47f0707a admin admin] PATH_INFO: `/v3/projects/229665c5468348078e180b08ac5f1a63/users/9cde4dfbcd28430e9af6a7a967c7bc00/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:23.993319 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f8eb7d-32ff-4dc8-bf2f-54fa47f0707a admin admin] RBAC: Authorizing `identity:create_grant(project_id=229665c5468348078e180b08ac5f1a63, user_id=9cde4dfbcd28430e9af6a7a967c7bc00, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:23.996480 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f8eb7d-32ff-4dc8-bf2f-54fa47f0707a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:24.010556 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1378/2773] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:23 2022] PUT /identity/v3/projects/229665c5468348078e180b08ac5f1a63/users/9cde4dfbcd28430e9af6a7a967c7bc00/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:24.015070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba9f87e-aeab-41a0-bda8-419619ef5278 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:24.025742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba9f87e-aeab-41a0-bda8-419619ef5278 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:24.034847 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:24.086166 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae51d509-bfa8-4c8d-9250-808964ba110a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:24.086836 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae51d509-bfa8-4c8d-9250-808964ba110a admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:24.087071 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae51d509-bfa8-4c8d-9250-808964ba110a admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:24.087352 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae51d509-bfa8-4c8d-9250-808964ba110a admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:24.088144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae51d509-bfa8-4c8d-9250-808964ba110a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:24.090378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae51d509-bfa8-4c8d-9250-808964ba110a admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:24.100519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1396/2774] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:24 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:24.104844 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f8eb7d-32ff-4dc8-bf2f-54fa47f0707a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:24.115895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f8eb7d-32ff-4dc8-bf2f-54fa47f0707a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:24.120992 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ad39c1-5d1f-4fe8-86b4-78caa0881d0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:24.121669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ad39c1-5d1f-4fe8-86b4-78caa0881d0b admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:24.121956 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ad39c1-5d1f-4fe8-86b4-78caa0881d0b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:24.122227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ad39c1-5d1f-4fe8-86b4-78caa0881d0b admin admin] PATH_INFO: `/v3/projects/229665c5468348078e180b08ac5f1a63/users/9cde4dfbcd28430e9af6a7a967c7bc00/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:24.124261 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ad39c1-5d1f-4fe8-86b4-78caa0881d0b admin admin] RBAC: Authorizing `identity:create_grant(project_id=229665c5468348078e180b08ac5f1a63, user_id=9cde4dfbcd28430e9af6a7a967c7bc00, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:24.126878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ad39c1-5d1f-4fe8-86b4-78caa0881d0b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:24.137472 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1379/2775] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:24 2022] PUT /identity/v3/projects/229665c5468348078e180b08ac5f1a63/users/9cde4dfbcd28430e9af6a7a967c7bc00/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:24.149250 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fe7bfa-5da2-4a64-bd15-1b6089e8bba6 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:24.149604 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fe7bfa-5da2-4a64-bd15-1b6089e8bba6 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:24.149887 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fe7bfa-5da2-4a64-bd15-1b6089e8bba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:24.180317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-94fe7bfa-5da2-4a64-bd15-1b6089e8bba6 None None] MFA Rules not processed for user `9cde4dfbcd28430e9af6a7a967c7bc00`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:24.181605 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-94fe7bfa-5da2-4a64-bd15-1b6089e8bba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:24.251405 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1397/2776] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:24 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:24.262262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ad39c1-5d1f-4fe8-86b4-78caa0881d0b admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:24.264003 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-74ad39c1-5d1f-4fe8-86b4-78caa0881d0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:24.287765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:24.326351 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ad39c1-5d1f-4fe8-86b4-78caa0881d0b admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:24.330274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afbd9ac-8552-4029-aa91-8c9b50b99640 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:24.330895 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afbd9ac-8552-4029-aa91-8c9b50b99640 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:24.331176 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afbd9ac-8552-4029-aa91-8c9b50b99640 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:24.331449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afbd9ac-8552-4029-aa91-8c9b50b99640 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:24.341169 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afbd9ac-8552-4029-aa91-8c9b50b99640 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:24.342582 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afbd9ac-8552-4029-aa91-8c9b50b99640 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:24.352156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1380/2777] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:24 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:24.856099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fe7bfa-5da2-4a64-bd15-1b6089e8bba6 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:24.857601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-94fe7bfa-5da2-4a64-bd15-1b6089e8bba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:24.869394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fe7bfa-5da2-4a64-bd15-1b6089e8bba6 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:24.873673 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9717b0e6-82ef-4385-8380-069569a93470 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:24.874279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9717b0e6-82ef-4385-8380-069569a93470 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:24.874568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9717b0e6-82ef-4385-8380-069569a93470 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:24.874846 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9717b0e6-82ef-4385-8380-069569a93470 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:24.930955 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9717b0e6-82ef-4385-8380-069569a93470 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:24.932294 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9717b0e6-82ef-4385-8380-069569a93470 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:24.941634 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1398/2778] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:24 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:26.157215 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afbd9ac-8552-4029-aa91-8c9b50b99640 service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:26.158668 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7afbd9ac-8552-4029-aa91-8c9b50b99640 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:26.226767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afbd9ac-8552-4029-aa91-8c9b50b99640 service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:26.231698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48767f7-2d1c-409a-9a5e-71ae5d4d6dbf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:26.232311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48767f7-2d1c-409a-9a5e-71ae5d4d6dbf service placement] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:26.232627 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48767f7-2d1c-409a-9a5e-71ae5d4d6dbf service placement] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:26.232896 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48767f7-2d1c-409a-9a5e-71ae5d4d6dbf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:26.234681 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f48767f7-2d1c-409a-9a5e-71ae5d4d6dbf service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:26.249028 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48767f7-2d1c-409a-9a5e-71ae5d4d6dbf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:26.250619 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48767f7-2d1c-409a-9a5e-71ae5d4d6dbf service placement] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:26.260815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1381/2779] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:26 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:32.106974 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9717b0e6-82ef-4385-8380-069569a93470 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:32.115531 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9717b0e6-82ef-4385-8380-069569a93470 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:32.119360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d6c45c-6cae-4ac9-8301-546b009a2bbf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:32.119982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6c45c-6cae-4ac9-8301-546b009a2bbf service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:32.120273 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6c45c-6cae-4ac9-8301-546b009a2bbf service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:32.120556 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d6c45c-6cae-4ac9-8301-546b009a2bbf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:32.151306 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:32.178378 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d6c45c-6cae-4ac9-8301-546b009a2bbf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:32.179746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6d6c45c-6cae-4ac9-8301-546b009a2bbf service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:32.189630 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1399/2780] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:32 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:34.274969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48767f7-2d1c-409a-9a5e-71ae5d4d6dbf service placement] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:34.335878 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48767f7-2d1c-409a-9a5e-71ae5d4d6dbf service placement] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:34.340430 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:34.341096 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:34.341384 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:34.341661 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] PATH_INFO: `/v3/users/77ee05a1b35d4fafa7ece47c44fa933f` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:34.342832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] RBAC: Authorizing `identity:delete_user(user_id=77ee05a1b35d4fafa7ece47c44fa933f)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:34.344880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:34.391030 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77ee05a1b35d4fafa7ece47c44fa933f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:34.398357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77ee05a1b35d4fafa7ece47c44fa933f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:34.399278 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77ee05a1b35d4fafa7ece47c44fa933f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:34.400069 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77ee05a1b35d4fafa7ece47c44fa933f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:34.402131 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:34.410452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77ee05a1b35d4fafa7ece47c44fa933f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:34.428804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1382/2781] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:34 2022] DELETE /identity/v3/users/77ee05a1b35d4fafa7ece47c44fa933f => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:34.433351 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f2fbe3-f4f0-4ea1-97c1-7968f98d3d4a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:34.433852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f2fbe3-f4f0-4ea1-97c1-7968f98d3d4a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:34.433962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f2fbe3-f4f0-4ea1-97c1-7968f98d3d4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:34.453587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-74f2fbe3-f4f0-4ea1-97c1-7968f98d3d4a None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:34.454827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-74f2fbe3-f4f0-4ea1-97c1-7968f98d3d4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:34.513069 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1400/2782] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:34.522880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:34.524291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:34.581116 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54d8ce2-6249-4156-b9b7-f6cd53f35bed admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:34.584861 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b635fa4-c317-4322-8493-74d9c9b46661 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:34.585483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b635fa4-c317-4322-8493-74d9c9b46661 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:34.585767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b635fa4-c317-4322-8493-74d9c9b46661 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:34.586045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b635fa4-c317-4322-8493-74d9c9b46661 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:34.595147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b635fa4-c317-4322-8493-74d9c9b46661 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:34.596583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b635fa4-c317-4322-8493-74d9c9b46661 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:34.606007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1383/2783] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:34.884222 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f2fbe3-f4f0-4ea1-97c1-7968f98d3d4a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:34.885754 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-74f2fbe3-f4f0-4ea1-97c1-7968f98d3d4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:34.897985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f2fbe3-f4f0-4ea1-97c1-7968f98d3d4a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:34.903080 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:34.903744 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:34.904027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:34.904337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] PATH_INFO: `/v3/projects/e8e3f3b6b7ea4c5db3e73497c47fed2b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:34.906238 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8e3f3b6b7ea4c5db3e73497c47fed2b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:34.908220 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:34.954588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8e3f3b6b7ea4c5db3e73497c47fed2b', 'request_id': 'req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:34.960882 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8e3f3b6b7ea4c5db3e73497c47fed2b', 'request_id': 'req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:34.969097 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8e3f3b6b7ea4c5db3e73497c47fed2b', 'request_id': 'req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:34.979253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] The token cache is being invalidate because project e8e3f3b6b7ea4c5db3e73497c47fed2b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:34.979547 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:34.980158 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:34.981662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1401/2784] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:34 2022] DELETE /identity/v3/projects/e8e3f3b6b7ea4c5db3e73497c47fed2b => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:34.986379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b635fa4-c317-4322-8493-74d9c9b46661 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:34.995701 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b635fa4-c317-4322-8493-74d9c9b46661 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:35.052555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:35.053142 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:35.053425 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:35.053698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] PATH_INFO: `/v3/users/ba2a01ae9ffa471b9f7001091dc4120e` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:35.054911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba2a01ae9ffa471b9f7001091dc4120e)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:35.056930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:35.104047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba2a01ae9ffa471b9f7001091dc4120e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.112206 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba2a01ae9ffa471b9f7001091dc4120e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.112925 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba2a01ae9ffa471b9f7001091dc4120e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.113578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba2a01ae9ffa471b9f7001091dc4120e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.120826 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba2a01ae9ffa471b9f7001091dc4120e'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.131656 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:35.141405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1384/2785] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:34 2022] DELETE /identity/v3/users/ba2a01ae9ffa471b9f7001091dc4120e => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:35.411925 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:35.413441 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:35.427941 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7882a92b-68fe-4056-b3eb-0bbbef0e9a8f admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:35.480701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:35.489087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:35.489784 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:35.490308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:35.490768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] PATH_INFO: `/v3/projects/2e38be77337b47979eaf910db1cf0c59` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:35.492279 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e38be77337b47979eaf910db1cf0c59)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:35.494275 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:35.541629 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e38be77337b47979eaf910db1cf0c59', 'request_id': 'req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.548883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e38be77337b47979eaf910db1cf0c59', 'request_id': 'req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.556381 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e38be77337b47979eaf910db1cf0c59', 'request_id': 'req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.566975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] The token cache is being invalidate because project 2e38be77337b47979eaf910db1cf0c59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:35.567266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.567898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.569438 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1402/2786] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:35 2022] DELETE /identity/v3/projects/2e38be77337b47979eaf910db1cf0c59 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:35.573433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:35.574864 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:35.588188 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44dac23b-45c7-4dae-82dd-cd4404ddc158 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:35.644427 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:35.645100 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:35.645390 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:35.645674 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] PATH_INFO: `/v3/users/9cde4dfbcd28430e9af6a7a967c7bc00` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:35.647145 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] RBAC: Authorizing `identity:delete_user(user_id=9cde4dfbcd28430e9af6a7a967c7bc00)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:35.648810 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:35.699703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9cde4dfbcd28430e9af6a7a967c7bc00'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.707644 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9cde4dfbcd28430e9af6a7a967c7bc00'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.708276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9cde4dfbcd28430e9af6a7a967c7bc00'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.708955 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9cde4dfbcd28430e9af6a7a967c7bc00'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.715818 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9cde4dfbcd28430e9af6a7a967c7bc00'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.733298 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1385/2787] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:35 2022] DELETE /identity/v3/users/9cde4dfbcd28430e9af6a7a967c7bc00 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:35.853502 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:35.854931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:35.868252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666a44d7-db1b-4cba-9f9e-7e367d5b37e3 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:35.938148 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:35.938795 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:35.939088 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:35.939369 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] PATH_INFO: `/v3/users/77ef2b72f9224ebab5340cff47c7eb5b` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:35.940590 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=77ef2b72f9224ebab5340cff47c7eb5b)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:35.942531 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:35.990066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '77ef2b72f9224ebab5340cff47c7eb5b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.990712 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '77ef2b72f9224ebab5340cff47c7eb5b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:35.991358 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '77ef2b72f9224ebab5340cff47c7eb5b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.006002 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '77ef2b72f9224ebab5340cff47c7eb5b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.011776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.013345 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-1af111aa-f145-4c77-85a2-a501fcd7bf0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '77ef2b72f9224ebab5340cff47c7eb5b'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.013916 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.030104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1403/2788] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:35 2022] DELETE /identity/v3/users/77ef2b72f9224ebab5340cff47c7eb5b => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:36.031134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648f6444-eca8-4dcc-be13-b9ec0894172d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.034474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ef304-d493-4e96-890d-18ae7725f171 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.034773 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ef304-d493-4e96-890d-18ae7725f171 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.035149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97ef304-d493-4e96-890d-18ae7725f171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.044485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:36.056580 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-c97ef304-d493-4e96-890d-18ae7725f171 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:36.058098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c97ef304-d493-4e96-890d-18ae7725f171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.096192 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.096837 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.097120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.097394 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] PATH_INFO: `/v3/projects/229665c5468348078e180b08ac5f1a63` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.098617 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] RBAC: Authorizing `identity:delete_project(project_id=229665c5468348078e180b08ac5f1a63)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.100452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.116824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:36.118917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1404/2789] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 05:00:36.130645 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97ef304-d493-4e96-890d-18ae7725f171 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.132860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c97ef304-d493-4e96-890d-18ae7725f171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.152225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '229665c5468348078e180b08ac5f1a63', 'request_id': 'req-a93b7211-09c4-457f-8229-c722cdb76746'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.160038 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '229665c5468348078e180b08ac5f1a63', 'request_id': 'req-a93b7211-09c4-457f-8229-c722cdb76746'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.166106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '229665c5468348078e180b08ac5f1a63', 'request_id': 'req-a93b7211-09c4-457f-8229-c722cdb76746'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.178141 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] The token cache is being invalidate because project 229665c5468348078e180b08ac5f1a63 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:36.178434 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.179113 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-a93b7211-09c4-457f-8229-c722cdb76746 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.180684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1386/2790] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:36 2022] DELETE /identity/v3/projects/229665c5468348078e180b08ac5f1a63 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:36.187614 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877e6be4-a0b3-4c2d-bc0c-eef6f30f0723 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.187894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877e6be4-a0b3-4c2d-bc0c-eef6f30f0723 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.188247 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877e6be4-a0b3-4c2d-bc0c-eef6f30f0723 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.195334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97ef304-d493-4e96-890d-18ae7725f171 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.210055 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-877e6be4-a0b3-4c2d-bc0c-eef6f30f0723 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:36.211287 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-877e6be4-a0b3-4c2d-bc0c-eef6f30f0723 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.251849 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891ab4be-fc74-4701-bcff-4ad5745c30ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.252485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891ab4be-fc74-4701-bcff-4ad5745c30ca service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.252810 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891ab4be-fc74-4701-bcff-4ad5745c30ca service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.253108 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891ab4be-fc74-4701-bcff-4ad5745c30ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.254921 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-891ab4be-fc74-4701-bcff-4ad5745c30ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.275881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1387/2791] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 05:00:36.285209 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164de0ac-835f-4842-92fb-7576edcb1012 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.285504 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164de0ac-835f-4842-92fb-7576edcb1012 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.285782 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164de0ac-835f-4842-92fb-7576edcb1012 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.306125 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-164de0ac-835f-4842-92fb-7576edcb1012 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:36.307334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-164de0ac-835f-4842-92fb-7576edcb1012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.316578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1388/2792] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:36.320249 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891ab4be-fc74-4701-bcff-4ad5745c30ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.320802 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164de0ac-835f-4842-92fb-7576edcb1012 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.321936 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891ab4be-fc74-4701-bcff-4ad5745c30ca service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.330029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164de0ac-835f-4842-92fb-7576edcb1012 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.332478 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1405/2793] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 05:00:36.334859 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1510a0b-b70c-4ff4-83fd-5b6696d387d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.335468 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1510a0b-b70c-4ff4-83fd-5b6696d387d8 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.335754 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1510a0b-b70c-4ff4-83fd-5b6696d387d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.336032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1510a0b-b70c-4ff4-83fd-5b6696d387d8 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.336770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1510a0b-b70c-4ff4-83fd-5b6696d387d8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 05:00:36.337115 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1510a0b-b70c-4ff4-83fd-5b6696d387d8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.338716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1510a0b-b70c-4ff4-83fd-5b6696d387d8 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.350854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1389/2794] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 05:00:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 05:00:36.355234 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891ab4be-fc74-4701-bcff-4ad5745c30ca service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.366150 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891ab4be-fc74-4701-bcff-4ad5745c30ca service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.370860 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ccaf3a-698e-4973-8a24-e7de6c0da087 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.371476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ccaf3a-698e-4973-8a24-e7de6c0da087 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.371766 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ccaf3a-698e-4973-8a24-e7de6c0da087 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.372048 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ccaf3a-698e-4973-8a24-e7de6c0da087 admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.372742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ccaf3a-698e-4973-8a24-e7de6c0da087 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.376039 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ccaf3a-698e-4973-8a24-e7de6c0da087 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.386772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:36.408435 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1406/2795] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/projects => generated 377 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:36.412727 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1510a0b-b70c-4ff4-83fd-5b6696d387d8 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.425225 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1510a0b-b70c-4ff4-83fd-5b6696d387d8 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.485792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dc8ad1-b29b-4cd2-8d67-c1f6de1cdaa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.486423 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dc8ad1-b29b-4cd2-8d67-c1f6de1cdaa0 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.486710 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dc8ad1-b29b-4cd2-8d67-c1f6de1cdaa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.487031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dc8ad1-b29b-4cd2-8d67-c1f6de1cdaa0 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.487705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dc8ad1-b29b-4cd2-8d67-c1f6de1cdaa0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.490106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dc8ad1-b29b-4cd2-8d67-c1f6de1cdaa0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.512620 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1390/2796] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/users => generated 289 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:36.516962 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ccaf3a-698e-4973-8a24-e7de6c0da087 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.527581 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ccaf3a-698e-4973-8a24-e7de6c0da087 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.532536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de35fc7-75db-4827-905e-2b0fbd3d81a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.533189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de35fc7-75db-4827-905e-2b0fbd3d81a2 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.533476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de35fc7-75db-4827-905e-2b0fbd3d81a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.533983 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de35fc7-75db-4827-905e-2b0fbd3d81a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.534501 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de35fc7-75db-4827-905e-2b0fbd3d81a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.536213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de35fc7-75db-4827-905e-2b0fbd3d81a2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.548231 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1407/2797] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 05:00:36.552284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dc8ad1-b29b-4cd2-8d67-c1f6de1cdaa0 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.563260 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dc8ad1-b29b-4cd2-8d67-c1f6de1cdaa0 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.568133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec2b91c-2182-416c-ab91-e1feb871c449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.568789 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec2b91c-2182-416c-ab91-e1feb871c449 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.569081 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec2b91c-2182-416c-ab91-e1feb871c449 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.569362 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec2b91c-2182-416c-ab91-e1feb871c449 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.570065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec2b91c-2182-416c-ab91-e1feb871c449 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.571677 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec2b91c-2182-416c-ab91-e1feb871c449 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.582686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1391/2798] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:36.586841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de35fc7-75db-4827-905e-2b0fbd3d81a2 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.597322 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de35fc7-75db-4827-905e-2b0fbd3d81a2 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.602171 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31245b8f-eff0-4641-8ddd-1ea2a87cc742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.602852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31245b8f-eff0-4641-8ddd-1ea2a87cc742 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.603236 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31245b8f-eff0-4641-8ddd-1ea2a87cc742 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.603593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31245b8f-eff0-4641-8ddd-1ea2a87cc742 admin admin] PATH_INFO: `/v3/projects/7a305c2ed41442d980f7f31e5cc9c0fc/users/cf30e06033224dabb654095dfb1a27f4/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.620327 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31245b8f-eff0-4641-8ddd-1ea2a87cc742 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a305c2ed41442d980f7f31e5cc9c0fc, user_id=cf30e06033224dabb654095dfb1a27f4, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.623251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31245b8f-eff0-4641-8ddd-1ea2a87cc742 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.630953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec2b91c-2182-416c-ab91-e1feb871c449 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.633180 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-8ec2b91c-2182-416c-ab91-e1feb871c449 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.642672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:36.643640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1408/2799] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:36 2022] PUT /identity/v3/projects/7a305c2ed41442d980f7f31e5cc9c0fc/users/cf30e06033224dabb654095dfb1a27f4/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:36.648291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec2b91c-2182-416c-ab91-e1feb871c449 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.657561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2385c05f-3312-4b00-9849-877fba4ed684 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.657918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2385c05f-3312-4b00-9849-877fba4ed684 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.658325 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2385c05f-3312-4b00-9849-877fba4ed684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.688657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2385c05f-3312-4b00-9849-877fba4ed684 None None] MFA Rules not processed for user `cf30e06033224dabb654095dfb1a27f4`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:36.689906 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2385c05f-3312-4b00-9849-877fba4ed684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.723151 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.723912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.724262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.724625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] PATH_INFO: `/v3/projects/9b4791b868c04418be1ba2c79ab278a9` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.725882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b4791b868c04418be1ba2c79ab278a9)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.727771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.754561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1409/2800] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 05:00:36.759149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2385c05f-3312-4b00-9849-877fba4ed684 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.770104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2385c05f-3312-4b00-9849-877fba4ed684 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.775356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b4791b868c04418be1ba2c79ab278a9', 'request_id': 'req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.784223 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b4791b868c04418be1ba2c79ab278a9', 'request_id': 'req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.790811 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b4791b868c04418be1ba2c79ab278a9', 'request_id': 'req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.803462 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] The token cache is being invalidate because project 9b4791b868c04418be1ba2c79ab278a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:36.803760 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.804436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-184ffb54-9aeb-4cca-a1ae-268a5ee43a60 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:36.806016 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1392/2801] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:36 2022] DELETE /identity/v3/projects/9b4791b868c04418be1ba2c79ab278a9 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:36.810822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0afc02-549a-43df-8b4d-f982e60a8aa9 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.811140 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0afc02-549a-43df-8b4d-f982e60a8aa9 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.811567 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0afc02-549a-43df-8b4d-f982e60a8aa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.831531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-ce0afc02-549a-43df-8b4d-f982e60a8aa9 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:36.832800 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ce0afc02-549a-43df-8b4d-f982e60a8aa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.837915 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d76b295-91d3-44af-b1f1-2541362b833d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.838667 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d76b295-91d3-44af-b1f1-2541362b833d admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.839046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d76b295-91d3-44af-b1f1-2541362b833d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.839329 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d76b295-91d3-44af-b1f1-2541362b833d admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.839992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d76b295-91d3-44af-b1f1-2541362b833d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.841029 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1393/2802] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 05:00:36.842079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d76b295-91d3-44af-b1f1-2541362b833d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.849405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c573e3-d20e-41de-9d81-b3b5f5ed4cad None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.849721 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c573e3-d20e-41de-9d81-b3b5f5ed4cad None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.849999 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c573e3-d20e-41de-9d81-b3b5f5ed4cad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.869573 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-24c573e3-d20e-41de-9d81-b3b5f5ed4cad None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:36.873561 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1410/2803] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/projects => generated 379 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:00:36.873761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-24c573e3-d20e-41de-9d81-b3b5f5ed4cad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.874759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d76b295-91d3-44af-b1f1-2541362b833d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.876388 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-5d76b295-91d3-44af-b1f1-2541362b833d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:36.909196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:36.933078 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1394/2804] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:36.937966 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c573e3-d20e-41de-9d81-b3b5f5ed4cad None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.945467 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d76b295-91d3-44af-b1f1-2541362b833d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.948120 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c573e3-d20e-41de-9d81-b3b5f5ed4cad None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.950529 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab976a9-5015-4d8d-8f9e-bcaea3219690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.951145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab976a9-5015-4d8d-8f9e-bcaea3219690 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.951432 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab976a9-5015-4d8d-8f9e-bcaea3219690 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.951710 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab976a9-5015-4d8d-8f9e-bcaea3219690 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.952651 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303ef9e-e6b9-46b7-969d-b5b992a9c468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.952834 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab976a9-5015-4d8d-8f9e-bcaea3219690 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.953255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303ef9e-e6b9-46b7-969d-b5b992a9c468 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.953672 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303ef9e-e6b9-46b7-969d-b5b992a9c468 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.953953 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e303ef9e-e6b9-46b7-969d-b5b992a9c468 admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.954645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e303ef9e-e6b9-46b7-969d-b5b992a9c468 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 05:00:36.954762 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dab976a9-5015-4d8d-8f9e-bcaea3219690 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.954950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e303ef9e-e6b9-46b7-969d-b5b992a9c468 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.956530 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e303ef9e-e6b9-46b7-969d-b5b992a9c468 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.970546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1395/2805] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 05:00:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 05:00:36.974831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303ef9e-e6b9-46b7-969d-b5b992a9c468 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.980689 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1411/2806] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/users => generated 290 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:36.981640 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab976a9-5015-4d8d-8f9e-bcaea3219690 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:36.985712 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e303ef9e-e6b9-46b7-969d-b5b992a9c468 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.990484 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e273ea8-95b3-4f05-9d4c-291f169458c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.991121 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e273ea8-95b3-4f05-9d4c-291f169458c3 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.991408 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e273ea8-95b3-4f05-9d4c-291f169458c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.991688 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e273ea8-95b3-4f05-9d4c-291f169458c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.992146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab976a9-5015-4d8d-8f9e-bcaea3219690 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:36.992352 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e273ea8-95b3-4f05-9d4c-291f169458c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:36.994285 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e273ea8-95b3-4f05-9d4c-291f169458c3 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:36.996796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8234c7-6c80-43d2-97d5-ed8af1472a65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:36.997396 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8234c7-6c80-43d2-97d5-ed8af1472a65 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:36.997682 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8234c7-6c80-43d2-97d5-ed8af1472a65 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:36.997960 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8234c7-6c80-43d2-97d5-ed8af1472a65 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:36.998730 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8234c7-6c80-43d2-97d5-ed8af1472a65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.000589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8234c7-6c80-43d2-97d5-ed8af1472a65 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.006805 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:37.011503 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1412/2807] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 05:00:37.015772 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8234c7-6c80-43d2-97d5-ed8af1472a65 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.025426 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8234c7-6c80-43d2-97d5-ed8af1472a65 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.030557 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1396/2808] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:36 2022] POST /identity/v3/projects => generated 371 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:00:37.031607 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e273ea8-95b3-4f05-9d4c-291f169458c3 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.087066 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedab709-7331-4afd-8234-58ca2b6005ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.087692 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedab709-7331-4afd-8234-58ca2b6005ab admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.087982 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedab709-7331-4afd-8234-58ca2b6005ab admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.088266 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedab709-7331-4afd-8234-58ca2b6005ab admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.089002 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedab709-7331-4afd-8234-58ca2b6005ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.090737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bedab709-7331-4afd-8234-58ca2b6005ab admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.094112 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e273ea8-95b3-4f05-9d4c-291f169458c3 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.098923 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a31db83-c6f4-4e8d-aca6-4bf9381a2ed9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.099540 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a31db83-c6f4-4e8d-aca6-4bf9381a2ed9 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.099823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a31db83-c6f4-4e8d-aca6-4bf9381a2ed9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.100101 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a31db83-c6f4-4e8d-aca6-4bf9381a2ed9 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.100798 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a31db83-c6f4-4e8d-aca6-4bf9381a2ed9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.101922 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1413/2809] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:37 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 05:00:37.105305 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a31db83-c6f4-4e8d-aca6-4bf9381a2ed9 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.106179 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedab709-7331-4afd-8234-58ca2b6005ab admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.116651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedab709-7331-4afd-8234-58ca2b6005ab admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.122072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261843d9-c660-43de-8ef1-b24dca2fdc33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.122668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261843d9-c660-43de-8ef1-b24dca2fdc33 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.122954 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261843d9-c660-43de-8ef1-b24dca2fdc33 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.123237 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261843d9-c660-43de-8ef1-b24dca2fdc33 admin admin] PATH_INFO: `/v3/projects/f0d6c187c3d54ca19cf9adb9861a1d64/users/433f61661f434444b8206d5b835db582/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.128803 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1397/2810] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:37 2022] POST /identity/v3/users => generated 286 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:00:37.129828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a31db83-c6f4-4e8d-aca6-4bf9381a2ed9 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.140433 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a31db83-c6f4-4e8d-aca6-4bf9381a2ed9 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.141136 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261843d9-c660-43de-8ef1-b24dca2fdc33 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0d6c187c3d54ca19cf9adb9861a1d64, user_id=433f61661f434444b8206d5b835db582, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.143767 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261843d9-c660-43de-8ef1-b24dca2fdc33 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.145276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c343aac9-13be-4810-806b-462e4534bfe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.145893 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343aac9-13be-4810-806b-462e4534bfe5 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.146205 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343aac9-13be-4810-806b-462e4534bfe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.146513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343aac9-13be-4810-806b-462e4534bfe5 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.147208 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c343aac9-13be-4810-806b-462e4534bfe5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.148911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c343aac9-13be-4810-806b-462e4534bfe5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.158853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1414/2811] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:37 2022] PUT /identity/v3/projects/f0d6c187c3d54ca19cf9adb9861a1d64/users/433f61661f434444b8206d5b835db582/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:37.159509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1398/2812] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 05:00:37.164654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261843d9-c660-43de-8ef1-b24dca2fdc33 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.170269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5330ac-6f5a-402d-a137-5dadeb56aa9b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.171227 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5330ac-6f5a-402d-a137-5dadeb56aa9b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.171506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5330ac-6f5a-402d-a137-5dadeb56aa9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.200536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6f5330ac-6f5a-402d-a137-5dadeb56aa9b None None] MFA Rules not processed for user `433f61661f434444b8206d5b835db582`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:37.201831 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6f5330ac-6f5a-402d-a137-5dadeb56aa9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:37.227694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261843d9-c660-43de-8ef1-b24dca2fdc33 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.232500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91c98c6-5089-4460-b90e-3d68ce93f244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.234141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c98c6-5089-4460-b90e-3d68ce93f244 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.234141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c98c6-5089-4460-b90e-3d68ce93f244 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.234363 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c98c6-5089-4460-b90e-3d68ce93f244 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.234995 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91c98c6-5089-4460-b90e-3d68ce93f244 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.236632 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91c98c6-5089-4460-b90e-3d68ce93f244 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.246872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1415/2813] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:37 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 05:00:37.251733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91c98c6-5089-4460-b90e-3d68ce93f244 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.261343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91c98c6-5089-4460-b90e-3d68ce93f244 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.266206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e511ab-9a27-40a8-beff-64462133cd1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.266851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e511ab-9a27-40a8-beff-64462133cd1b admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.267147 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e511ab-9a27-40a8-beff-64462133cd1b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.267429 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e511ab-9a27-40a8-beff-64462133cd1b admin admin] PATH_INFO: `/v3/projects/b586a0d507ca497589a504f9e2d44574/users/ead3628943944bbeabaa2e6a80b99f74/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.268699 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1399/2814] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:37 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:37.273158 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5330ac-6f5a-402d-a137-5dadeb56aa9b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.279582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:37.283087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e511ab-9a27-40a8-beff-64462133cd1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b586a0d507ca497589a504f9e2d44574, user_id=ead3628943944bbeabaa2e6a80b99f74, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.284175 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5330ac-6f5a-402d-a137-5dadeb56aa9b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.286138 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e511ab-9a27-40a8-beff-64462133cd1b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.288950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55d8b1-f708-40a4-8421-070e16a06739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.289560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55d8b1-f708-40a4-8421-070e16a06739 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.289847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55d8b1-f708-40a4-8421-070e16a06739 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.290243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55d8b1-f708-40a4-8421-070e16a06739 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.290905 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d55d8b1-f708-40a4-8421-070e16a06739 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.292789 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d55d8b1-f708-40a4-8421-070e16a06739 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.301343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1416/2815] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:37 2022] PUT /identity/v3/projects/b586a0d507ca497589a504f9e2d44574/users/ead3628943944bbeabaa2e6a80b99f74/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:37.306205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa17df98-e654-439f-a20d-4537232de285 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.306500 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa17df98-e654-439f-a20d-4537232de285 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.306786 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa17df98-e654-439f-a20d-4537232de285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.326571 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1400/2816] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:37 2022] POST /identity/v3/projects => generated 379 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:37.326796 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-fa17df98-e654-439f-a20d-4537232de285 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:37.327771 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55d8b1-f708-40a4-8421-070e16a06739 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.327899 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fa17df98-e654-439f-a20d-4537232de285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:37.369335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:37.387416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1417/2817] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 29 05:00:37.392595 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55d8b1-f708-40a4-8421-070e16a06739 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.397260 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa17df98-e654-439f-a20d-4537232de285 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.397509 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ddae6f-1f73-428f-bdbc-4b5e22c7f0c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.398133 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ddae6f-1f73-428f-bdbc-4b5e22c7f0c4 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.398419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ddae6f-1f73-428f-bdbc-4b5e22c7f0c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.398704 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ddae6f-1f73-428f-bdbc-4b5e22c7f0c4 admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.398805 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fa17df98-e654-439f-a20d-4537232de285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:37.399379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ddae6f-1f73-428f-bdbc-4b5e22c7f0c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.401381 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ddae6f-1f73-428f-bdbc-4b5e22c7f0c4 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.425530 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1401/2818] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:37 2022] POST /identity/v3/users => generated 296 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:00:37.429432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ddae6f-1f73-428f-bdbc-4b5e22c7f0c4 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.439943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ddae6f-1f73-428f-bdbc-4b5e22c7f0c4 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.444685 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8118b666-3c42-42e4-bec2-1081b6758d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.445296 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8118b666-3c42-42e4-bec2-1081b6758d1b admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.445579 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8118b666-3c42-42e4-bec2-1081b6758d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.445857 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8118b666-3c42-42e4-bec2-1081b6758d1b admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.446555 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8118b666-3c42-42e4-bec2-1081b6758d1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.448144 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8118b666-3c42-42e4-bec2-1081b6758d1b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.457351 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa17df98-e654-439f-a20d-4537232de285 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.459007 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1402/2819] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:37.461385 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9692bb29-ad7f-4696-91ce-6a6e76e89540 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.462004 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9692bb29-ad7f-4696-91ce-6a6e76e89540 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.462295 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9692bb29-ad7f-4696-91ce-6a6e76e89540 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.462573 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9692bb29-ad7f-4696-91ce-6a6e76e89540 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.462996 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8118b666-3c42-42e4-bec2-1081b6758d1b admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.472992 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9692bb29-ad7f-4696-91ce-6a6e76e89540 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.473669 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8118b666-3c42-42e4-bec2-1081b6758d1b admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.474682 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9692bb29-ad7f-4696-91ce-6a6e76e89540 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.478345 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f438ee-e325-43fe-b9f4-dd8389977d93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.479129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f438ee-e325-43fe-b9f4-dd8389977d93 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.479417 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f438ee-e325-43fe-b9f4-dd8389977d93 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.479703 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f438ee-e325-43fe-b9f4-dd8389977d93 admin admin] PATH_INFO: `/v3/projects/cd6f6da166ab46838f7a6f509fa2b5bd/users/ec8a194e0fd64e9cad63762d5033e282/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.485347 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1418/2820] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 05:00:37.495398 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f438ee-e325-43fe-b9f4-dd8389977d93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd6f6da166ab46838f7a6f509fa2b5bd, user_id=ec8a194e0fd64e9cad63762d5033e282, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.498268 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f438ee-e325-43fe-b9f4-dd8389977d93 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.512546 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1403/2821] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:37 2022] PUT /identity/v3/projects/cd6f6da166ab46838f7a6f509fa2b5bd/users/ec8a194e0fd64e9cad63762d5033e282/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:37.516588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9692bb29-ad7f-4696-91ce-6a6e76e89540 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.526578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9692bb29-ad7f-4696-91ce-6a6e76e89540 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.587589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89d67df-83f1-47b3-b122-14588e8dd592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.588354 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89d67df-83f1-47b3-b122-14588e8dd592 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.588765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89d67df-83f1-47b3-b122-14588e8dd592 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.589164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89d67df-83f1-47b3-b122-14588e8dd592 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.589873 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89d67df-83f1-47b3-b122-14588e8dd592 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.591479 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89d67df-83f1-47b3-b122-14588e8dd592 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.605343 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1419/2822] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:37 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:37.606578 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f438ee-e325-43fe-b9f4-dd8389977d93 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.618174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f438ee-e325-43fe-b9f4-dd8389977d93 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.622995 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766a178e-513e-42b2-951f-cabc3a0c9fdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.623723 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766a178e-513e-42b2-951f-cabc3a0c9fdc admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.624095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766a178e-513e-42b2-951f-cabc3a0c9fdc admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.624375 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766a178e-513e-42b2-951f-cabc3a0c9fdc admin admin] PATH_INFO: `/v3/projects/cd6f6da166ab46838f7a6f509fa2b5bd/users/ec8a194e0fd64e9cad63762d5033e282/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.626373 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766a178e-513e-42b2-951f-cabc3a0c9fdc admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd6f6da166ab46838f7a6f509fa2b5bd, user_id=ec8a194e0fd64e9cad63762d5033e282, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.629024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766a178e-513e-42b2-951f-cabc3a0c9fdc admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.647407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1404/2823] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:37 2022] PUT /identity/v3/projects/cd6f6da166ab46838f7a6f509fa2b5bd/users/ec8a194e0fd64e9cad63762d5033e282/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:37.658344 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6aa9c73-ddcb-433f-b267-51c7c79b08b5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.658663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6aa9c73-ddcb-433f-b267-51c7c79b08b5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.658944 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6aa9c73-ddcb-433f-b267-51c7c79b08b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.695377 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-d6aa9c73-ddcb-433f-b267-51c7c79b08b5 None None] MFA Rules not processed for user `ec8a194e0fd64e9cad63762d5033e282`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:37.696568 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d6aa9c73-ddcb-433f-b267-51c7c79b08b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:37.719280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:37.784226 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1420/2824] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:37 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:37.796400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766a178e-513e-42b2-951f-cabc3a0c9fdc admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.798255 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-766a178e-513e-42b2-951f-cabc3a0c9fdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:37.870306 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766a178e-513e-42b2-951f-cabc3a0c9fdc admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.874210 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173ef92a-3b1e-4491-a77a-f7e3d285ca61 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.874954 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173ef92a-3b1e-4491-a77a-f7e3d285ca61 service cinder] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.875241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173ef92a-3b1e-4491-a77a-f7e3d285ca61 service cinder] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.875645 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-173ef92a-3b1e-4491-a77a-f7e3d285ca61 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.888165 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173ef92a-3b1e-4491-a77a-f7e3d285ca61 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:37.890828 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-173ef92a-3b1e-4491-a77a-f7e3d285ca61 service cinder] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:37.903040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1405/2825] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:37 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:37.947911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6aa9c73-ddcb-433f-b267-51c7c79b08b5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:37.949413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-d6aa9c73-ddcb-433f-b267-51c7c79b08b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:37.962361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6aa9c73-ddcb-433f-b267-51c7c79b08b5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:37.966662 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee02e93-75b3-4b5f-93a4-3be4e0b69283 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:37.969009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee02e93-75b3-4b5f-93a4-3be4e0b69283 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:37.969009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee02e93-75b3-4b5f-93a4-3be4e0b69283 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:37.969009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee02e93-75b3-4b5f-93a4-3be4e0b69283 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:37.970918 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-8ee02e93-75b3-4b5f-93a4-3be4e0b69283 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:38.033759 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee02e93-75b3-4b5f-93a4-3be4e0b69283 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:38.035241 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ee02e93-75b3-4b5f-93a4-3be4e0b69283 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:38.045474 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1421/2826] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:37 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:41.168714 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173ef92a-3b1e-4491-a77a-f7e3d285ca61 service cinder] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:41.258548 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-173ef92a-3b1e-4491-a77a-f7e3d285ca61 service cinder] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:41.263226 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:41.263889 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:41.264336 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:41.264835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] PATH_INFO: `/v3/users/cf30e06033224dabb654095dfb1a27f4` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:41.266158 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf30e06033224dabb654095dfb1a27f4)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:41.268106 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:41.295392 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=91,keystone:INSERT=5,keystone:DELETE=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:41.328302 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf30e06033224dabb654095dfb1a27f4'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:41.341114 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf30e06033224dabb654095dfb1a27f4'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:41.341875 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf30e06033224dabb654095dfb1a27f4'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:41.343153 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf30e06033224dabb654095dfb1a27f4'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:41.352525 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf30e06033224dabb654095dfb1a27f4'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:41.373032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1406/2827] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:41 2022] DELETE /identity/v3/users/cf30e06033224dabb654095dfb1a27f4 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:41.377693 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f29eaea-b8ba-4172-bc4a-bd7ea644ee7c None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:41.378151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f29eaea-b8ba-4172-bc4a-bd7ea644ee7c None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:41.378248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f29eaea-b8ba-4172-bc4a-bd7ea644ee7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:41.422442 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-2f29eaea-b8ba-4172-bc4a-bd7ea644ee7c None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:41.423985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-2f29eaea-b8ba-4172-bc4a-bd7ea644ee7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:41.500208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1422/2828] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:41.510675 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:41.512122 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:41.583598 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78eef11a-0a85-452d-a54f-1836011004b7 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:41.587335 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e254789-ac33-4a94-bf49-904ae0d53522 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:41.587969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e254789-ac33-4a94-bf49-904ae0d53522 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:41.588256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e254789-ac33-4a94-bf49-904ae0d53522 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:41.588536 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e254789-ac33-4a94-bf49-904ae0d53522 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:41.599419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e254789-ac33-4a94-bf49-904ae0d53522 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:41.601061 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e254789-ac33-4a94-bf49-904ae0d53522 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:41.613767 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1407/2829] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:41.916829 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a54c297-0d4d-4a08-b938-78514638fe3a None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:41.917129 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a54c297-0d4d-4a08-b938-78514638fe3a None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:41.923663 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a54c297-0d4d-4a08-b938-78514638fe3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:41.979652 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-3a54c297-0d4d-4a08-b938-78514638fe3a None None] MFA Rules not processed for user `ead3628943944bbeabaa2e6a80b99f74`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:41.980904 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3a54c297-0d4d-4a08-b938-78514638fe3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:42.034519 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e254789-ac33-4a94-bf49-904ae0d53522 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:42.036739 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-1e254789-ac33-4a94-bf49-904ae0d53522 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:42.062562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=100 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:42.066150 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e254789-ac33-4a94-bf49-904ae0d53522 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:42.090975 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1423/2830] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:41 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:42.092801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:42.093542 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:42.094506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:42.094506 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] PATH_INFO: `/v3/projects/7a305c2ed41442d980f7f31e5cc9c0fc` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:42.095524 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a305c2ed41442d980f7f31e5cc9c0fc)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:42.096735 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a54c297-0d4d-4a08-b938-78514638fe3a None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:42.097496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:42.098296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3a54c297-0d4d-4a08-b938-78514638fe3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:42.112649 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a54c297-0d4d-4a08-b938-78514638fe3a None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:42.117574 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cebf98-81af-4080-9f46-6aed9ea3c03b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:42.118198 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cebf98-81af-4080-9f46-6aed9ea3c03b admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:42.118482 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cebf98-81af-4080-9f46-6aed9ea3c03b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:42.118760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cebf98-81af-4080-9f46-6aed9ea3c03b admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:42.119427 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cebf98-81af-4080-9f46-6aed9ea3c03b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:42.121462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cebf98-81af-4080-9f46-6aed9ea3c03b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:42.159257 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1424/2831] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:42 2022] POST /identity/v3/projects => generated 371 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:42.163027 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cebf98-81af-4080-9f46-6aed9ea3c03b admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:42.165031 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a305c2ed41442d980f7f31e5cc9c0fc', 'request_id': 'req-d33e350b-384a-4c99-85eb-99a515f7db83'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.177071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a305c2ed41442d980f7f31e5cc9c0fc', 'request_id': 'req-d33e350b-384a-4c99-85eb-99a515f7db83'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.185456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a305c2ed41442d980f7f31e5cc9c0fc', 'request_id': 'req-d33e350b-384a-4c99-85eb-99a515f7db83'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.199063 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] The token cache is being invalidate because project 7a305c2ed41442d980f7f31e5cc9c0fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:42.199636 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.200456 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.203432 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1408/2832] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:42 2022] DELETE /identity/v3/projects/7a305c2ed41442d980f7f31e5cc9c0fc => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:42.208139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:42.210137 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:42.248344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:42.277225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cebf98-81af-4080-9f46-6aed9ea3c03b admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:42.283020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f4c8ba-e279-4b89-9371-36c509f8dd6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:42.283727 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f4c8ba-e279-4b89-9371-36c509f8dd6e admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:42.284024 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f4c8ba-e279-4b89-9371-36c509f8dd6e admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:42.284307 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f4c8ba-e279-4b89-9371-36c509f8dd6e admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:42.285018 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f4c8ba-e279-4b89-9371-36c509f8dd6e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:42.287035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f4c8ba-e279-4b89-9371-36c509f8dd6e admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:42.303117 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33e350b-384a-4c99-85eb-99a515f7db83 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:42.309443 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:42.310341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1425/2833] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:42 2022] POST /identity/v3/users => generated 292 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:42.310585 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:42.310881 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:42.311749 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] PATH_INFO: `/v3/users/433f61661f434444b8206d5b835db582` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:42.313095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] RBAC: Authorizing `identity:delete_user(user_id=433f61661f434444b8206d5b835db582)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:42.315203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f4c8ba-e279-4b89-9371-36c509f8dd6e admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:42.315379 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:42.317091 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b1f4c8ba-e279-4b89-9371-36c509f8dd6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:42.330448 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f4c8ba-e279-4b89-9371-36c509f8dd6e admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:42.336213 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e672ce-ed19-47d1-9107-70cb31b5791d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:42.340078 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e672ce-ed19-47d1-9107-70cb31b5791d admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:42.340366 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e672ce-ed19-47d1-9107-70cb31b5791d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:42.340905 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e672ce-ed19-47d1-9107-70cb31b5791d admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:42.342599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e672ce-ed19-47d1-9107-70cb31b5791d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:42.344392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e672ce-ed19-47d1-9107-70cb31b5791d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:42.355648 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1426/2834] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:42 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 05:00:42.359906 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e672ce-ed19-47d1-9107-70cb31b5791d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:42.370718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e672ce-ed19-47d1-9107-70cb31b5791d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:42.372401 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '433f61661f434444b8206d5b835db582'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.377104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2fda04-7e9a-4447-8d79-0458c525baf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:42.378019 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2fda04-7e9a-4447-8d79-0458c525baf3 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:42.378595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2fda04-7e9a-4447-8d79-0458c525baf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:42.378595 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2fda04-7e9a-4447-8d79-0458c525baf3 admin admin] PATH_INFO: `/v3/projects/3cb92a2970e6455bb09be55ef55c19e6/users/3debda601db84e08a5cf0484f510bdb0/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:42.381820 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '433f61661f434444b8206d5b835db582'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.382621 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '433f61661f434444b8206d5b835db582'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.383406 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '433f61661f434444b8206d5b835db582'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.394243 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '433f61661f434444b8206d5b835db582'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.397103 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2fda04-7e9a-4447-8d79-0458c525baf3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cb92a2970e6455bb09be55ef55c19e6, user_id=3debda601db84e08a5cf0484f510bdb0, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:42.399845 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2fda04-7e9a-4447-8d79-0458c525baf3 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:42.418601 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1427/2835] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:42 2022] PUT /identity/v3/projects/3cb92a2970e6455bb09be55ef55c19e6/users/3debda601db84e08a5cf0484f510bdb0/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:42.420067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1409/2836] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:42 2022] DELETE /identity/v3/users/433f61661f434444b8206d5b835db582 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:42.426742 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2fda04-7e9a-4447-8d79-0458c525baf3 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:42.428737 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fa2fda04-7e9a-4447-8d79-0458c525baf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:42.507565 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2fda04-7e9a-4447-8d79-0458c525baf3 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:42.531364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:42.578760 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabdfbbc-5c82-4ad5-a682-8e8d9064fd21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:42.579606 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabdfbbc-5c82-4ad5-a682-8e8d9064fd21 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:42.579959 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabdfbbc-5c82-4ad5-a682-8e8d9064fd21 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:42.580367 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabdfbbc-5c82-4ad5-a682-8e8d9064fd21 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:42.581265 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabdfbbc-5c82-4ad5-a682-8e8d9064fd21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:42.583120 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabdfbbc-5c82-4ad5-a682-8e8d9064fd21 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:42.594942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1428/2837] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:42 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:42.599376 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:42.610327 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19902e29-eec0-46b9-ad3a-21ad9df4ccce admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:42.614887 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8d4f0a-6359-4443-8c07-cba50a79dff5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:42.615684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8d4f0a-6359-4443-8c07-cba50a79dff5 admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:42.616693 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8d4f0a-6359-4443-8c07-cba50a79dff5 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:42.616980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8d4f0a-6359-4443-8c07-cba50a79dff5 admin admin] PATH_INFO: `/v3/projects/3cb92a2970e6455bb09be55ef55c19e6/users/3debda601db84e08a5cf0484f510bdb0/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:42.619002 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8d4f0a-6359-4443-8c07-cba50a79dff5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cb92a2970e6455bb09be55ef55c19e6, user_id=3debda601db84e08a5cf0484f510bdb0, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:42.621756 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8d4f0a-6359-4443-8c07-cba50a79dff5 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:42.633024 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1410/2838] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:42 2022] PUT /identity/v3/projects/3cb92a2970e6455bb09be55ef55c19e6/users/3debda601db84e08a5cf0484f510bdb0/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:42.716430 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabdfbbc-5c82-4ad5-a682-8e8d9064fd21 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:42.719842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-dabdfbbc-5c82-4ad5-a682-8e8d9064fd21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:42.736722 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabdfbbc-5c82-4ad5-a682-8e8d9064fd21 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:42.814216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:42.815010 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:42.815320 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:42.815643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] PATH_INFO: `/v3/projects/f0d6c187c3d54ca19cf9adb9861a1d64` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:42.816978 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0d6c187c3d54ca19cf9adb9861a1d64)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:42.818886 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:42.880957 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0d6c187c3d54ca19cf9adb9861a1d64', 'request_id': 'req-a35e3a3b-6248-4033-9bdc-1e17eded0957'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.888371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0d6c187c3d54ca19cf9adb9861a1d64', 'request_id': 'req-a35e3a3b-6248-4033-9bdc-1e17eded0957'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.898900 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0d6c187c3d54ca19cf9adb9861a1d64', 'request_id': 'req-a35e3a3b-6248-4033-9bdc-1e17eded0957'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.914587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] The token cache is being invalidate because project f0d6c187c3d54ca19cf9adb9861a1d64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:42.915037 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.916102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:42.917931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1429/2839] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:42 2022] DELETE /identity/v3/projects/f0d6c187c3d54ca19cf9adb9861a1d64 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:42.922882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8d4f0a-6359-4443-8c07-cba50a79dff5 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:42.925526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-7c8d4f0a-6359-4443-8c07-cba50a79dff5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:42.939156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8d4f0a-6359-4443-8c07-cba50a79dff5 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:43.017134 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:43.017952 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:43.018303 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:43.018652 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] PATH_INFO: `/v3/users/ec8a194e0fd64e9cad63762d5033e282` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:43.019939 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec8a194e0fd64e9cad63762d5033e282)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:43.022761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:43.080904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:43.094067 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec8a194e0fd64e9cad63762d5033e282'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:43.105182 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec8a194e0fd64e9cad63762d5033e282'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:43.105914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec8a194e0fd64e9cad63762d5033e282'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:43.106684 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec8a194e0fd64e9cad63762d5033e282'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:43.117246 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-8332c18c-d215-4a51-89b7-6748cbcf8602 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec8a194e0fd64e9cad63762d5033e282'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:43.137606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1411/2840] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:00:42 2022] DELETE /identity/v3/users/ec8a194e0fd64e9cad63762d5033e282 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:43.449636 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:43.462837 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35e3a3b-6248-4033-9bdc-1e17eded0957 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:43.522929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:43.536070 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:43.536911 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:43.537130 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:43.538102 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] PATH_INFO: `/v3/projects/cd6f6da166ab46838f7a6f509fa2b5bd` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:43.539928 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd6f6da166ab46838f7a6f509fa2b5bd)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:43.542551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:43.597552 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd6f6da166ab46838f7a6f509fa2b5bd', 'request_id': 'req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:43.606068 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd6f6da166ab46838f7a6f509fa2b5bd', 'request_id': 'req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:43.615423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd6f6da166ab46838f7a6f509fa2b5bd', 'request_id': 'req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:43.627587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] The token cache is being invalidate because project cd6f6da166ab46838f7a6f509fa2b5bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:00:43.627987 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:43.628563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-62fc1e9a-c877-45d4-ba80-7ae8ba54fa64 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:00:43.634716 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1430/2841] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:00:43 2022] DELETE /identity/v3/projects/cd6f6da166ab46838f7a6f509fa2b5bd => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:00:43.636304 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03214219-cf99-4c21-a7e4-1ffd12322b97 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:43.636790 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03214219-cf99-4c21-a7e4-1ffd12322b97 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:43.636994 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03214219-cf99-4c21-a7e4-1ffd12322b97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:43.660269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-03214219-cf99-4c21-a7e4-1ffd12322b97 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:43.665606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-03214219-cf99-4c21-a7e4-1ffd12322b97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:43.735419 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1412/2842] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:43.744182 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0db705c-8de8-4fcc-b72b-880c270bb4ea None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:43.744657 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0db705c-8de8-4fcc-b72b-880c270bb4ea None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:43.745085 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0db705c-8de8-4fcc-b72b-880c270bb4ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:43.769471 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-e0db705c-8de8-4fcc-b72b-880c270bb4ea None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:43.771714 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e0db705c-8de8-4fcc-b72b-880c270bb4ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:43.785973 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1431/2843] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:43.790315 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03214219-cf99-4c21-a7e4-1ffd12322b97 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:43.805615 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03214219-cf99-4c21-a7e4-1ffd12322b97 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:43.812970 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afe3425-a485-4f71-af14-84f7082a074b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:43.813606 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe3425-a485-4f71-af14-84f7082a074b admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:43.813894 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe3425-a485-4f71-af14-84f7082a074b admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:43.814173 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe3425-a485-4f71-af14-84f7082a074b admin admin] PATH_INFO: `/v3/domains` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:43.815051 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afe3425-a485-4f71-af14-84f7082a074b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92189) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 05:00:43.815366 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afe3425-a485-4f71-af14-84f7082a074b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:43.817047 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afe3425-a485-4f71-af14-84f7082a074b admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:43.833990 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1413/2844] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 05:00:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 29 05:00:43.838269 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0db705c-8de8-4fcc-b72b-880c270bb4ea None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:43.850514 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0db705c-8de8-4fcc-b72b-880c270bb4ea None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:43.857122 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cd0153-c185-4635-8b83-ba5f4b07253b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:43.858853 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd0153-c185-4635-8b83-ba5f4b07253b admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:43.859146 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd0153-c185-4635-8b83-ba5f4b07253b admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:43.859618 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd0153-c185-4635-8b83-ba5f4b07253b admin admin] PATH_INFO: `/v3/projects` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:43.860419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cd0153-c185-4635-8b83-ba5f4b07253b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:43.863101 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cd0153-c185-4635-8b83-ba5f4b07253b admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:43.898748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1432/2845] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:00:43 2022] POST /identity/v3/projects => generated 389 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:43.902813 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afe3425-a485-4f71-af14-84f7082a074b admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:43.922011 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afe3425-a485-4f71-af14-84f7082a074b admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:44.017565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fabe8e-1b54-4977-9501-84f3ca851f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:44.018294 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fabe8e-1b54-4977-9501-84f3ca851f6e admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:44.018648 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fabe8e-1b54-4977-9501-84f3ca851f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:44.018998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fabe8e-1b54-4977-9501-84f3ca851f6e admin admin] PATH_INFO: `/v3/users` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:44.019744 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fabe8e-1b54-4977-9501-84f3ca851f6e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:44.021973 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fabe8e-1b54-4977-9501-84f3ca851f6e admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:44.055159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cd0153-c185-4635-8b83-ba5f4b07253b admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:44.060822 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1414/2846] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:00:43 2022] POST /identity/v3/users => generated 295 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:00:44.068961 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cd0153-c185-4635-8b83-ba5f4b07253b admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:44.081517 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3a23ab-9560-4832-9c68-da99108e53b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:44.082162 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3a23ab-9560-4832-9c68-da99108e53b2 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:44.082508 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3a23ab-9560-4832-9c68-da99108e53b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:44.082801 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3a23ab-9560-4832-9c68-da99108e53b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:44.083516 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3a23ab-9560-4832-9c68-da99108e53b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:44.085543 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3a23ab-9560-4832-9c68-da99108e53b2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:44.096684 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1433/2847] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:44 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:44.100584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fabe8e-1b54-4977-9501-84f3ca851f6e admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:44.115488 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fabe8e-1b54-4977-9501-84f3ca851f6e admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:44.125449 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab6227b-cbba-4ee2-9752-27750dff15d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:44.126599 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab6227b-cbba-4ee2-9752-27750dff15d2 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:44.127759 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab6227b-cbba-4ee2-9752-27750dff15d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:44.128288 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab6227b-cbba-4ee2-9752-27750dff15d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:44.129599 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab6227b-cbba-4ee2-9752-27750dff15d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:44.132127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab6227b-cbba-4ee2-9752-27750dff15d2 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:44.145633 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1415/2848] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:00:44 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:00:44.149785 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3a23ab-9560-4832-9c68-da99108e53b2 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:44.163050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3a23ab-9560-4832-9c68-da99108e53b2 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:44.169842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ce5107-3fa6-4265-a83f-a0462dc09351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:44.171651 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ce5107-3fa6-4265-a83f-a0462dc09351 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:44.172280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ce5107-3fa6-4265-a83f-a0462dc09351 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:44.172813 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ce5107-3fa6-4265-a83f-a0462dc09351 admin admin] PATH_INFO: `/v3/projects/8def6fc92fee49ecbef3bda26dde1ba3/users/1f2e9984d7104b8db0278a1647b2b88f/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:44.201443 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ce5107-3fa6-4265-a83f-a0462dc09351 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8def6fc92fee49ecbef3bda26dde1ba3, user_id=1f2e9984d7104b8db0278a1647b2b88f, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:44.205708 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ce5107-3fa6-4265-a83f-a0462dc09351 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:44.236096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1434/2849] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:00:44 2022] PUT /identity/v3/projects/8def6fc92fee49ecbef3bda26dde1ba3/users/1f2e9984d7104b8db0278a1647b2b88f/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:00:44.244249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2763baf-d393-4e6e-ae5d-28567a905dff None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:44.244249 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2763baf-d393-4e6e-ae5d-28567a905dff None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:44.244529 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2763baf-d393-4e6e-ae5d-28567a905dff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:44.279395 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-c2763baf-d393-4e6e-ae5d-28567a905dff None None] MFA Rules not processed for user `1f2e9984d7104b8db0278a1647b2b88f`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:44.281313 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-c2763baf-d393-4e6e-ae5d-28567a905dff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:44.286944 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:44.369538 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1416/2850] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:44 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:44.379449 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ce5107-3fa6-4265-a83f-a0462dc09351 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:44.381330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-06ce5107-3fa6-4265-a83f-a0462dc09351 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:44.449229 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ce5107-3fa6-4265-a83f-a0462dc09351 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:44.453643 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d81fd4f-16ec-4c02-99c4-06395ddba27c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:44.454389 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d81fd4f-16ec-4c02-99c4-06395ddba27c service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:44.454771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d81fd4f-16ec-4c02-99c4-06395ddba27c service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:44.455154 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d81fd4f-16ec-4c02-99c4-06395ddba27c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:44.465167 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d81fd4f-16ec-4c02-99c4-06395ddba27c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:44.466706 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d81fd4f-16ec-4c02-99c4-06395ddba27c service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:44.477539 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1435/2851] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:44 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:00:47.558276 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8993525-4790-41d3-a436-f730f38d8861 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:47.558676 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8993525-4790-41d3-a436-f730f38d8861 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:47.558846 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8993525-4790-41d3-a436-f730f38d8861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:47.600892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-a8993525-4790-41d3-a436-f730f38d8861 None None] MFA Rules not processed for user `3debda601db84e08a5cf0484f510bdb0`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:47.602413 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a8993525-4790-41d3-a436-f730f38d8861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:47.693334 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1417/2852] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:47 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:47.710864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d81fd4f-16ec-4c02-99c4-06395ddba27c service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:00:47.712298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-3d81fd4f-16ec-4c02-99c4-06395ddba27c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:47.736912 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:47.793851 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d81fd4f-16ec-4c02-99c4-06395ddba27c service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:00:47.800929 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7854992-0d2b-4962-8d2b-ae180f27544b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:00:47.801558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7854992-0d2b-4962-8d2b-ae180f27544b service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:47.801852 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7854992-0d2b-4962-8d2b-ae180f27544b service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:47.802137 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7854992-0d2b-4962-8d2b-ae180f27544b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:47.803976 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-c7854992-0d2b-4962-8d2b-ae180f27544b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:47.862707 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7854992-0d2b-4962-8d2b-ae180f27544b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:00:47.864301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7854992-0d2b-4962-8d2b-ae180f27544b service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:00:47.873189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1436/2853] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:00:47 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 05:00:48.912022 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d90971-4f25-447a-8a39-73f74b58f34e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:00:48.912908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d90971-4f25-447a-8a39-73f74b58f34e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:00:48.913212 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d90971-4f25-447a-8a39-73f74b58f34e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:00:48.933933 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-78d90971-4f25-447a-8a39-73f74b58f34e None None] MFA Rules not processed for user `ead3628943944bbeabaa2e6a80b99f74`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:00:48.935129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-78d90971-4f25-447a-8a39-73f74b58f34e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:00:48.942630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1418/2854] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:00:48 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:00:57.869149 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=47 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:00:58.927400 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=61 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:01:10.741339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7854992-0d2b-4962-8d2b-ae180f27544b service nova] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:10.802301 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7854992-0d2b-4962-8d2b-ae180f27544b service nova] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:10.806763 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:10.807391 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:10.807675 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:10.807958 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] PATH_INFO: `/v3/users/1f2e9984d7104b8db0278a1647b2b88f` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:10.809159 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f2e9984d7104b8db0278a1647b2b88f)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:10.811062 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:10.856475 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f2e9984d7104b8db0278a1647b2b88f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:01:10.857160 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f2e9984d7104b8db0278a1647b2b88f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:01:10.857782 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f2e9984d7104b8db0278a1647b2b88f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:01:10.870505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f2e9984d7104b8db0278a1647b2b88f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:01:10.877582 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f2e9984d7104b8db0278a1647b2b88f'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:01:10.890752 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1437/2855] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:01:10 2022] DELETE /identity/v3/users/1f2e9984d7104b8db0278a1647b2b88f => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:01:10.895310 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39468a97-6a44-4125-8c66-8b1597bf502e None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:10.895832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39468a97-6a44-4125-8c66-8b1597bf502e None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:10.895942 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39468a97-6a44-4125-8c66-8b1597bf502e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:10.914792 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-39468a97-6a44-4125-8c66-8b1597bf502e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:01:10.915912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-39468a97-6a44-4125-8c66-8b1597bf502e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:10.972539 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1419/2856] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:01:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:01:10.982205 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:10.983653 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:11.041452 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048d34eb-cc69-4d08-a45d-a14ef8ad30e7 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:11.045404 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c8dce54-7863-49ba-b31f-39f1c814dde4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:11.046021 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8dce54-7863-49ba-b31f-39f1c814dde4 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.046308 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8dce54-7863-49ba-b31f-39f1c814dde4 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.046588 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c8dce54-7863-49ba-b31f-39f1c814dde4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.056546 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8dce54-7863-49ba-b31f-39f1c814dde4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:11.058031 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c8dce54-7863-49ba-b31f-39f1c814dde4 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:11.067841 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1438/2857] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:01:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:01:11.333383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39468a97-6a44-4125-8c66-8b1597bf502e None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:11.334812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-39468a97-6a44-4125-8c66-8b1597bf502e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:11.349274 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39468a97-6a44-4125-8c66-8b1597bf502e None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:11.354124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:11.354783 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.355071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.355354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] PATH_INFO: `/v3/projects/8def6fc92fee49ecbef3bda26dde1ba3` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.356531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8def6fc92fee49ecbef3bda26dde1ba3)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:11.358496 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:11.405062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8def6fc92fee49ecbef3bda26dde1ba3', 'request_id': 'req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:01:11.413904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8def6fc92fee49ecbef3bda26dde1ba3', 'request_id': 'req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:01:11.419650 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8def6fc92fee49ecbef3bda26dde1ba3', 'request_id': 'req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:01:11.430568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] The token cache is being invalidate because project 8def6fc92fee49ecbef3bda26dde1ba3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:01:11.430855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:01:11.431483 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-39c345fe-213c-4e75-ae4c-60d4ad5f37c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:01:11.433057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1420/2858] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:01:11 2022] DELETE /identity/v3/projects/8def6fc92fee49ecbef3bda26dde1ba3 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:01:11.438462 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b8a754-9346-422e-b6cb-ff81c3e2b7b1 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.438823 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b8a754-9346-422e-b6cb-ff81c3e2b7b1 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.439558 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b8a754-9346-422e-b6cb-ff81c3e2b7b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.460468 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-34b8a754-9346-422e-b6cb-ff81c3e2b7b1 None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:01:11.461732 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-34b8a754-9346-422e-b6cb-ff81c3e2b7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:11.479496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:01:11.524409 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1439/2859] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:01:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:01:11.533565 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098c8ac6-50fa-4739-acd2-49ad53d81fff None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.533860 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098c8ac6-50fa-4739-acd2-49ad53d81fff None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.534174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098c8ac6-50fa-4739-acd2-49ad53d81fff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.555060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-098c8ac6-50fa-4739-acd2-49ad53d81fff None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:01:11.556267 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-098c8ac6-50fa-4739-acd2-49ad53d81fff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:11.565071 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1421/2860] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:01:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:01:11.569306 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b8a754-9346-422e-b6cb-ff81c3e2b7b1 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:11.578842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b8a754-9346-422e-b6cb-ff81c3e2b7b1 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:11.584186 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5d0185-7ac5-4e97-a38b-ff3754abe515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:11.584827 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5d0185-7ac5-4e97-a38b-ff3754abe515 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.585116 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5d0185-7ac5-4e97-a38b-ff3754abe515 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.585394 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5d0185-7ac5-4e97-a38b-ff3754abe515 admin admin] PATH_INFO: `/v3/domains` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.586089 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5d0185-7ac5-4e97-a38b-ff3754abe515 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=92190) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 29 05:01:11.586400 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5d0185-7ac5-4e97-a38b-ff3754abe515 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:11.588043 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5d0185-7ac5-4e97-a38b-ff3754abe515 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:11.600155 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1440/2861] 10.222.0.40 () {58 vars in 1153 bytes} [Fri Jul 29 05:01:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 29 05:01:11.604284 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098c8ac6-50fa-4739-acd2-49ad53d81fff None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:11.614350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098c8ac6-50fa-4739-acd2-49ad53d81fff None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:11.618984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a693e33-a8fe-44b7-8268-b0627f160e79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:11.619589 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a693e33-a8fe-44b7-8268-b0627f160e79 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.619871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a693e33-a8fe-44b7-8268-b0627f160e79 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.620147 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a693e33-a8fe-44b7-8268-b0627f160e79 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.620848 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a693e33-a8fe-44b7-8268-b0627f160e79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:11.622763 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a693e33-a8fe-44b7-8268-b0627f160e79 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:11.649035 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1422/2862] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:01:11 2022] POST /identity/v3/projects => generated 377 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:01:11.653126 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5d0185-7ac5-4e97-a38b-ff3754abe515 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:11.664403 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5d0185-7ac5-4e97-a38b-ff3754abe515 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:11.722713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32011e08-5a00-4128-b1f2-e618ff4feebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:11.723255 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32011e08-5a00-4128-b1f2-e618ff4feebb admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.723541 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32011e08-5a00-4128-b1f2-e618ff4feebb admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.723821 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32011e08-5a00-4128-b1f2-e618ff4feebb admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.724493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32011e08-5a00-4128-b1f2-e618ff4feebb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:11.726437 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32011e08-5a00-4128-b1f2-e618ff4feebb admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:11.747416 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1441/2863] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:01:11 2022] POST /identity/v3/users => generated 289 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:01:11.752095 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a693e33-a8fe-44b7-8268-b0627f160e79 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:11.762426 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a693e33-a8fe-44b7-8268-b0627f160e79 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:11.767512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117e62d0-a967-4947-bce5-f3ae9a875068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:11.768124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e62d0-a967-4947-bce5-f3ae9a875068 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.768405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e62d0-a967-4947-bce5-f3ae9a875068 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.768707 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117e62d0-a967-4947-bce5-f3ae9a875068 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.769393 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117e62d0-a967-4947-bce5-f3ae9a875068 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:11.770972 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117e62d0-a967-4947-bce5-f3ae9a875068 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:11.780908 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1423/2864] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:01:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:01:11.784936 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32011e08-5a00-4128-b1f2-e618ff4feebb admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:11.795332 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32011e08-5a00-4128-b1f2-e618ff4feebb admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:11.800006 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53366a0a-ff3f-4157-bdfe-50db60bfae88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:11.800628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53366a0a-ff3f-4157-bdfe-50db60bfae88 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.800927 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53366a0a-ff3f-4157-bdfe-50db60bfae88 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.801208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53366a0a-ff3f-4157-bdfe-50db60bfae88 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.802026 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53366a0a-ff3f-4157-bdfe-50db60bfae88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:11.803641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53366a0a-ff3f-4157-bdfe-50db60bfae88 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:11.814765 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1442/2865] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:01:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 29 05:01:11.818849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117e62d0-a967-4947-bce5-f3ae9a875068 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:11.828686 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117e62d0-a967-4947-bce5-f3ae9a875068 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:11.833982 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b2ce4f-2d60-4c28-8f20-245962558cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:11.834630 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b2ce4f-2d60-4c28-8f20-245962558cac admin admin] REQUEST_METHOD: `PUT` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.834911 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b2ce4f-2d60-4c28-8f20-245962558cac admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.835241 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b2ce4f-2d60-4c28-8f20-245962558cac admin admin] PATH_INFO: `/v3/projects/e10af2e6edcd485caf51326fdc9c17fc/users/6fd1b82319424fb8bd7e50bc0cb00037/roles/9e77a799dd224c1480f51a54f3aed529` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.850168 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b2ce4f-2d60-4c28-8f20-245962558cac admin admin] RBAC: Authorizing `identity:create_grant(project_id=e10af2e6edcd485caf51326fdc9c17fc, user_id=6fd1b82319424fb8bd7e50bc0cb00037, role_id=9e77a799dd224c1480f51a54f3aed529)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:11.852854 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b2ce4f-2d60-4c28-8f20-245962558cac admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:11.865568 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1424/2866] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:01:11 2022] PUT /identity/v3/projects/e10af2e6edcd485caf51326fdc9c17fc/users/6fd1b82319424fb8bd7e50bc0cb00037/roles/9e77a799dd224c1480f51a54f3aed529 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:01:11.870450 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15dba15-b59a-4f52-8da9-256d32d5525e None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:11.870748 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15dba15-b59a-4f52-8da9-256d32d5525e None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:11.871034 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15dba15-b59a-4f52-8da9-256d32d5525e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:11.890330 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-e15dba15-b59a-4f52-8da9-256d32d5525e None None] MFA Rules not processed for user `cfc9fe47ed4740a0aaddec545dbaca1a`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:01:11.892704 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-e15dba15-b59a-4f52-8da9-256d32d5525e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:11.952668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1443/2867] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:01:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:01:11.962179 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b2ce4f-2d60-4c28-8f20-245962558cac admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:11.963912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f7b2ce4f-2d60-4c28-8f20-245962558cac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:11.986680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:01:12.023861 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b2ce4f-2d60-4c28-8f20-245962558cac admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:12.027575 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735c5e37-81b3-45dc-a4e2-a8193d5b2461 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:12.028190 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735c5e37-81b3-45dc-a4e2-a8193d5b2461 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:12.028476 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735c5e37-81b3-45dc-a4e2-a8193d5b2461 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:12.028793 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735c5e37-81b3-45dc-a4e2-a8193d5b2461 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:12.038580 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735c5e37-81b3-45dc-a4e2-a8193d5b2461 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:12.039998 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735c5e37-81b3-45dc-a4e2-a8193d5b2461 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:12.049882 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1425/2868] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:01:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:01:15.774278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4573795-a6c3-4f5e-931f-c67087523db5 None None] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:15.774889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4573795-a6c3-4f5e-931f-c67087523db5 None None] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:15.774889 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4573795-a6c3-4f5e-931f-c67087523db5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:15.814251 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.auth.core [None req-a4573795-a6c3-4f5e-931f-c67087523db5 None None] MFA Rules not processed for user `6fd1b82319424fb8bd7e50bc0cb00037`. Rule list: `[]` (Enabled: `True`). {{(pid=92190) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:01:15.815562 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-a4573795-a6c3-4f5e-931f-c67087523db5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:15.836728 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:01:15.892253 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1444/2869] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:01:15 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:01:15.896849 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735c5e37-81b3-45dc-a4e2-a8193d5b2461 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:15.913493 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735c5e37-81b3-45dc-a4e2-a8193d5b2461 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:15.920256 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea2cc7c-1d8b-4dbd-9902-0c76b5ab0060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:15.922208 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea2cc7c-1d8b-4dbd-9902-0c76b5ab0060 admin admin] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:15.922583 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea2cc7c-1d8b-4dbd-9902-0c76b5ab0060 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:15.922776 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea2cc7c-1d8b-4dbd-9902-0c76b5ab0060 admin admin] PATH_INFO: `/v3/projects` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:15.924189 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea2cc7c-1d8b-4dbd-9902-0c76b5ab0060 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:15.926695 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea2cc7c-1d8b-4dbd-9902-0c76b5ab0060 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:15.985129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1426/2870] 10.222.0.40 () {60 vars in 1153 bytes} [Fri Jul 29 05:01:15 2022] POST /identity/v3/projects => generated 377 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 29 05:01:15.989072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4573795-a6c3-4f5e-931f-c67087523db5 None None] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:16.005337 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4573795-a6c3-4f5e-931f-c67087523db5 None None] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:16.086971 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf955b6-35f9-4866-8ad4-ac9802123100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:16.087608 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf955b6-35f9-4866-8ad4-ac9802123100 admin admin] REQUEST_METHOD: `POST` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:16.087892 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf955b6-35f9-4866-8ad4-ac9802123100 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:16.088326 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf955b6-35f9-4866-8ad4-ac9802123100 admin admin] PATH_INFO: `/v3/users` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:16.089046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf955b6-35f9-4866-8ad4-ac9802123100 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:16.091013 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf955b6-35f9-4866-8ad4-ac9802123100 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:16.125335 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1445/2871] 10.222.0.40 () {60 vars in 1144 bytes} [Fri Jul 29 05:01:15 2022] POST /identity/v3/users => generated 295 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 29 05:01:16.129010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea2cc7c-1d8b-4dbd-9902-0c76b5ab0060 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:16.140812 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea2cc7c-1d8b-4dbd-9902-0c76b5ab0060 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:16.146060 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5ea7ec-63ed-4396-85b5-d2467048b185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:16.146770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5ea7ec-63ed-4396-85b5-d2467048b185 admin admin] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:16.147064 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5ea7ec-63ed-4396-85b5-d2467048b185 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:16.147365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5ea7ec-63ed-4396-85b5-d2467048b185 admin admin] PATH_INFO: `/v3/roles` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:16.148049 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5ea7ec-63ed-4396-85b5-d2467048b185 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:16.149732 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5ea7ec-63ed-4396-85b5-d2467048b185 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:16.162855 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1427/2872] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:01:16 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:01:16.167009 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf955b6-35f9-4866-8ad4-ac9802123100 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:16.172281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5ea7ec-63ed-4396-85b5-d2467048b185 admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:16.179278 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf955b6-35f9-4866-8ad4-ac9802123100 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:16.184963 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd1d195-4a84-4b4d-b9c7-502b0da45f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:16.186228 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1d195-4a84-4b4d-b9c7-502b0da45f9d admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:16.186525 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1d195-4a84-4b4d-b9c7-502b0da45f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:16.186811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd1d195-4a84-4b4d-b9c7-502b0da45f9d admin admin] PATH_INFO: `/v3/projects/b28758e3ebd64fc9864d134a33085838/users/c58cae9e447143d48edf759a92204e4f/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:16.206399 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd1d195-4a84-4b4d-b9c7-502b0da45f9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b28758e3ebd64fc9864d134a33085838, user_id=c58cae9e447143d48edf759a92204e4f, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:16.209551 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd1d195-4a84-4b4d-b9c7-502b0da45f9d admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:16.232872 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1446/2873] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:01:16 2022] PUT /identity/v3/projects/b28758e3ebd64fc9864d134a33085838/users/c58cae9e447143d48edf759a92204e4f/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:01:16.233980 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd1d195-4a84-4b4d-b9c7-502b0da45f9d admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:16.252930 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5ea7ec-63ed-4396-85b5-d2467048b185 admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:16.311193 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986aa434-bf15-45b5-80e7-ee9bde4f62ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:16.312059 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986aa434-bf15-45b5-80e7-ee9bde4f62ad service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:16.312409 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986aa434-bf15-45b5-80e7-ee9bde4f62ad service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:16.312550 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd1d195-4a84-4b4d-b9c7-502b0da45f9d admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:16.312680 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986aa434-bf15-45b5-80e7-ee9bde4f62ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:16.315013 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-986aa434-bf15-45b5-80e7-ee9bde4f62ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:16.319669 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b3abd6-7ed5-4406-842d-4dacea5926e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:16.320361 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b3abd6-7ed5-4406-842d-4dacea5926e9 admin admin] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:16.320749 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b3abd6-7ed5-4406-842d-4dacea5926e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:16.321144 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b3abd6-7ed5-4406-842d-4dacea5926e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:16.321942 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b3abd6-7ed5-4406-842d-4dacea5926e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:16.324280 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b3abd6-7ed5-4406-842d-4dacea5926e9 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:16.325595 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986aa434-bf15-45b5-80e7-ee9bde4f62ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:16.327130 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-986aa434-bf15-45b5-80e7-ee9bde4f62ad service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:16.337248 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1447/2874] 10.222.0.40 () {58 vars in 1122 bytes} [Fri Jul 29 05:01:16 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 29 05:01:16.337726 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1428/2875] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:01:16 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 05:01:16.341029 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b3abd6-7ed5-4406-842d-4dacea5926e9 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:16.353599 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b3abd6-7ed5-4406-842d-4dacea5926e9 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:16.361511 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d9c726-70bd-4c08-bebe-30fac7c91ff2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:16.362413 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d9c726-70bd-4c08-bebe-30fac7c91ff2 admin admin] REQUEST_METHOD: `PUT` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:16.362883 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d9c726-70bd-4c08-bebe-30fac7c91ff2 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:16.363172 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d9c726-70bd-4c08-bebe-30fac7c91ff2 admin admin] PATH_INFO: `/v3/projects/b28758e3ebd64fc9864d134a33085838/users/c58cae9e447143d48edf759a92204e4f/roles/ca6111d7d86f4ac29c63630ce5250fd1` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:16.365298 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d9c726-70bd-4c08-bebe-30fac7c91ff2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b28758e3ebd64fc9864d134a33085838, user_id=c58cae9e447143d48edf759a92204e4f, role_id=ca6111d7d86f4ac29c63630ce5250fd1)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:16.368023 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d9c726-70bd-4c08-bebe-30fac7c91ff2 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:16.387252 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1448/2876] 10.222.0.40 () {60 vars in 1483 bytes} [Fri Jul 29 05:01:16 2022] PUT /identity/v3/projects/b28758e3ebd64fc9864d134a33085838/users/c58cae9e447143d48edf759a92204e4f/roles/ca6111d7d86f4ac29c63630ce5250fd1 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:01:22.678923 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da6884d-322f-4984-9ff9-009a29ad3d8b None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:22.679429 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da6884d-322f-4984-9ff9-009a29ad3d8b None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:22.679531 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da6884d-322f-4984-9ff9-009a29ad3d8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:22.710647 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-6da6884d-322f-4984-9ff9-009a29ad3d8b None None] MFA Rules not processed for user `c58cae9e447143d48edf759a92204e4f`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:01:22.711993 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-6da6884d-322f-4984-9ff9-009a29ad3d8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:22.755959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:01:22.789354 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1429/2877] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:01:22 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:01:22.804698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d9c726-70bd-4c08-bebe-30fac7c91ff2 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:22.806578 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-f6d9c726-70bd-4c08-bebe-30fac7c91ff2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:22.819499 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:01:22.885917 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d9c726-70bd-4c08-bebe-30fac7c91ff2 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:22.891471 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae129f88-5e37-41ff-8fcc-3ff54d9c3478 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:22.892678 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae129f88-5e37-41ff-8fcc-3ff54d9c3478 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:22.892965 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae129f88-5e37-41ff-8fcc-3ff54d9c3478 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:22.893233 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae129f88-5e37-41ff-8fcc-3ff54d9c3478 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:22.949092 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae129f88-5e37-41ff-8fcc-3ff54d9c3478 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:22.950617 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae129f88-5e37-41ff-8fcc-3ff54d9c3478 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:22.960382 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1449/2878] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:01:22 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:01:32.785604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=12 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:01:32.956311 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=53 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:01:35.448709 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da6884d-322f-4984-9ff9-009a29ad3d8b None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:35.502929 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da6884d-322f-4984-9ff9-009a29ad3d8b None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:35.506622 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c08c5cd-9d19-4d18-88fe-43db7e092c58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:35.507240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c08c5cd-9d19-4d18-88fe-43db7e092c58 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:35.507535 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c08c5cd-9d19-4d18-88fe-43db7e092c58 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:35.507815 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c08c5cd-9d19-4d18-88fe-43db7e092c58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:35.509618 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-3c08c5cd-9d19-4d18-88fe-43db7e092c58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:35.571281 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c08c5cd-9d19-4d18-88fe-43db7e092c58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:35.572746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c08c5cd-9d19-4d18-88fe-43db7e092c58 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:35.581969 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1430/2879] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:01:35 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:01:37.145724 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae129f88-5e37-41ff-8fcc-3ff54d9c3478 service nova] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:01:37.157745 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae129f88-5e37-41ff-8fcc-3ff54d9c3478 service nova] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:01:37.164560 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ece414-013d-48df-a8f9-30bb1ab3b8cf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:01:37.165203 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ece414-013d-48df-a8f9-30bb1ab3b8cf service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:37.165493 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ece414-013d-48df-a8f9-30bb1ab3b8cf service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:37.165775 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ece414-013d-48df-a8f9-30bb1ab3b8cf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:37.167536 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-05ece414-013d-48df-a8f9-30bb1ab3b8cf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:37.182907 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ece414-013d-48df-a8f9-30bb1ab3b8cf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:01:37.184655 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ece414-013d-48df-a8f9-30bb1ab3b8cf service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:01:37.196842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1450/2880] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:01:37 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:01:44.664006 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7112f78-7d9e-4604-9960-9a5000ff0470 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:01:44.664441 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7112f78-7d9e-4604-9960-9a5000ff0470 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:01:44.664562 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7112f78-7d9e-4604-9960-9a5000ff0470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:01:44.683884 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-f7112f78-7d9e-4604-9960-9a5000ff0470 None None] MFA Rules not processed for user `6fd1b82319424fb8bd7e50bc0cb00037`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:01:44.685129 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-f7112f78-7d9e-4604-9960-9a5000ff0470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:01:44.692032 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1431/2881] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:01:44 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:01:47.191701 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:01:54.678522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=53 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:02:03.517721 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ece414-013d-48df-a8f9-30bb1ab3b8cf service nova] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:02:03.519175 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-05ece414-013d-48df-a8f9-30bb1ab3b8cf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:02:03.531498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ece414-013d-48df-a8f9-30bb1ab3b8cf service nova] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:02:03.535099 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b54572d-53f8-4be0-9128-5fef792f9fc2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:02:03.535713 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b54572d-53f8-4be0-9128-5fef792f9fc2 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:02:03.535999 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b54572d-53f8-4be0-9128-5fef792f9fc2 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:02:03.536277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b54572d-53f8-4be0-9128-5fef792f9fc2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:02:03.545466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b54572d-53f8-4be0-9128-5fef792f9fc2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:02:03.546884 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b54572d-53f8-4be0-9128-5fef792f9fc2 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:02:03.556668 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1451/2882] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:02:03 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:02:13.552060 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:03:35.845489 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7112f78-7d9e-4604-9960-9a5000ff0470 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:03:35.855806 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7112f78-7d9e-4604-9960-9a5000ff0470 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:03:35.859361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38994a4-45b4-4fdd-ac68-0ed55c243c95 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:03:35.860008 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38994a4-45b4-4fdd-ac68-0ed55c243c95 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:03:35.860322 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38994a4-45b4-4fdd-ac68-0ed55c243c95 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:03:35.860581 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38994a4-45b4-4fdd-ac68-0ed55c243c95 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:03:35.862365 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b38994a4-45b4-4fdd-ac68-0ed55c243c95 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:03:35.871770 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38994a4-45b4-4fdd-ac68-0ed55c243c95 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:03:35.873311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38994a4-45b4-4fdd-ac68-0ed55c243c95 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:03:35.882127 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1432/2883] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:03:35 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:03:45.878384 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=6 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:03:53.331880 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b54572d-53f8-4be0-9128-5fef792f9fc2 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:03:53.340771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b54572d-53f8-4be0-9128-5fef792f9fc2 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:03:53.344477 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21b2cb7-b3d1-4782-a241-c21550df8281 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:03:53.345123 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21b2cb7-b3d1-4782-a241-c21550df8281 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:03:53.345419 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21b2cb7-b3d1-4782-a241-c21550df8281 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:03:53.345698 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21b2cb7-b3d1-4782-a241-c21550df8281 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:03:53.348087 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-b21b2cb7-b3d1-4782-a241-c21550df8281 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:03:53.362756 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21b2cb7-b3d1-4782-a241-c21550df8281 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:03:53.364371 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21b2cb7-b3d1-4782-a241-c21550df8281 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:03:53.375360 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1452/2884] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:03:53 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:03:54.562840 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38994a4-45b4-4fdd-ac68-0ed55c243c95 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:03:54.572236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38994a4-45b4-4fdd-ac68-0ed55c243c95 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:03:54.575902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9de69e-a936-4f99-808f-3a1dd6414fcc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:03:54.576514 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9de69e-a936-4f99-808f-3a1dd6414fcc service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:03:54.576829 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9de69e-a936-4f99-808f-3a1dd6414fcc service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:03:54.577110 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9de69e-a936-4f99-808f-3a1dd6414fcc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:03:54.578876 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4a9de69e-a936-4f99-808f-3a1dd6414fcc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:03:54.592046 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9de69e-a936-4f99-808f-3a1dd6414fcc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:03:54.593503 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9de69e-a936-4f99-808f-3a1dd6414fcc service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:03:54.604873 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1433/2885] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:03:54 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:04:03.370418 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:04:04.600042 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=6 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:04:07.851423 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21b2cb7-b3d1-4782-a241-c21550df8281 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:04:07.860310 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21b2cb7-b3d1-4782-a241-c21550df8281 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:04:07.864210 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef61afbf-8885-4229-a568-b6058d139ef2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:04:07.864864 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef61afbf-8885-4229-a568-b6058d139ef2 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:04:07.865151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef61afbf-8885-4229-a568-b6058d139ef2 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:04:07.865431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef61afbf-8885-4229-a568-b6058d139ef2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:04:07.867145 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ef61afbf-8885-4229-a568-b6058d139ef2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:04:07.877254 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef61afbf-8885-4229-a568-b6058d139ef2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:04:07.878969 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef61afbf-8885-4229-a568-b6058d139ef2 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:04:07.888141 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1453/2886] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:04:07 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:04:08.981295 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9de69e-a936-4f99-808f-3a1dd6414fcc service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:04:08.983344 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4a9de69e-a936-4f99-808f-3a1dd6414fcc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:04:08.998746 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9de69e-a936-4f99-808f-3a1dd6414fcc service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:04:09.004984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fa8dba-a01b-4cae-ab69-1a4f8bc31f99 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:04:09.005613 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fa8dba-a01b-4cae-ab69-1a4f8bc31f99 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:04:09.005902 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fa8dba-a01b-4cae-ab69-1a4f8bc31f99 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:04:09.006183 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fa8dba-a01b-4cae-ab69-1a4f8bc31f99 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:04:09.007980 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-20fa8dba-a01b-4cae-ab69-1a4f8bc31f99 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:04:09.009822 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef61afbf-8885-4229-a568-b6058d139ef2 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:04:09.018895 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef61afbf-8885-4229-a568-b6058d139ef2 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:04:09.020959 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fa8dba-a01b-4cae-ab69-1a4f8bc31f99 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:04:09.022522 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fa8dba-a01b-4cae-ab69-1a4f8bc31f99 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:04:09.023032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec558ab5-8a87-4d9c-a08b-7aed607862a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:04:09.023723 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec558ab5-8a87-4d9c-a08b-7aed607862a7 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:04:09.024084 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec558ab5-8a87-4d9c-a08b-7aed607862a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:04:09.024431 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec558ab5-8a87-4d9c-a08b-7aed607862a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:04:09.026739 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ec558ab5-8a87-4d9c-a08b-7aed607862a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:04:09.042842 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec558ab5-8a87-4d9c-a08b-7aed607862a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:04:09.044665 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec558ab5-8a87-4d9c-a08b-7aed607862a7 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:04:09.044816 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1434/2887] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:04:08 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:04:09.059300 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1454/2888] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:04:09 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:04:09.126291 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fa8dba-a01b-4cae-ab69-1a4f8bc31f99 service nova] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:04:09.127689 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-20fa8dba-a01b-4cae-ab69-1a4f8bc31f99 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:04:09.203755 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fa8dba-a01b-4cae-ab69-1a4f8bc31f99 service nova] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:04:09.209350 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f0e8b9-0e22-44c1-a68c-d449b8e9c192 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:04:09.209967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0e8b9-0e22-44c1-a68c-d449b8e9c192 service placement] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:04:09.210248 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0e8b9-0e22-44c1-a68c-d449b8e9c192 service placement] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:04:09.210523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0e8b9-0e22-44c1-a68c-d449b8e9c192 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:04:09.212292 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-b8f0e8b9-0e22-44c1-a68c-d449b8e9c192 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:04:09.225222 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f0e8b9-0e22-44c1-a68c-d449b8e9c192 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:04:09.226830 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f0e8b9-0e22-44c1-a68c-d449b8e9c192 service placement] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:04:09.236964 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1435/2889] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:04:09 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:04:19.053334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=12 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:04:19.232505 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=38 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:04:24.039705 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec558ab5-8a87-4d9c-a08b-7aed607862a7 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:04:24.054985 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec558ab5-8a87-4d9c-a08b-7aed607862a7 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:04:24.063592 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cd9b2f-538e-43e1-8a85-9c80b19971b4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:04:24.064324 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd9b2f-538e-43e1-8a85-9c80b19971b4 service nova] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:04:24.064736 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd9b2f-538e-43e1-8a85-9c80b19971b4 service nova] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:04:24.065139 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87cd9b2f-538e-43e1-8a85-9c80b19971b4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:04:24.067939 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-87cd9b2f-538e-43e1-8a85-9c80b19971b4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:04:24.086259 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cd9b2f-538e-43e1-8a85-9c80b19971b4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:04:24.087878 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87cd9b2f-538e-43e1-8a85-9c80b19971b4 service nova] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:04:24.104802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1455/2890] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:04:24 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:04:24.976057 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7f7531-3360-400a-acc0-da8b04668978 None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:04:24.976683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7f7531-3360-400a-acc0-da8b04668978 None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:04:24.976683 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7f7531-3360-400a-acc0-da8b04668978 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:04:24.995640 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-0f7f7531-3360-400a-acc0-da8b04668978 None None] MFA Rules not processed for user `6fd1b82319424fb8bd7e50bc0cb00037`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:04:24.996886 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-0f7f7531-3360-400a-acc0-da8b04668978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:04:25.003758 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1436/2891] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:04:24 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:04:34.095498 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:04:34.990808 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:05:31.169072 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cd9b2f-538e-43e1-8a85-9c80b19971b4 service nova] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:05:31.227579 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87cd9b2f-538e-43e1-8a85-9c80b19971b4 service nova] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:05:31.232694 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba921226-5112-4c19-a0da-2151cfd54558 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:05:31.233305 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba921226-5112-4c19-a0da-2151cfd54558 service placement] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:05:31.233587 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba921226-5112-4c19-a0da-2151cfd54558 service placement] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:05:31.233898 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba921226-5112-4c19-a0da-2151cfd54558 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:05:31.235641 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-ba921226-5112-4c19-a0da-2151cfd54558 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:05:31.269398 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba921226-5112-4c19-a0da-2151cfd54558 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:05:31.270979 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba921226-5112-4c19-a0da-2151cfd54558 service placement] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:05:31.281107 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1456/2892] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:05:31 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:05:41.276768 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=36 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:06:03.014236 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7f7531-3360-400a-acc0-da8b04668978 None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:06:03.038566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7f7531-3360-400a-acc0-da8b04668978 None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:06:03.042218 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91b6f14-b58c-4b4e-8d9c-e0dce8255295 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:06:03.042835 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b6f14-b58c-4b4e-8d9c-e0dce8255295 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:06:03.043124 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b6f14-b58c-4b4e-8d9c-e0dce8255295 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:06:03.043407 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91b6f14-b58c-4b4e-8d9c-e0dce8255295 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:06:03.045181 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a91b6f14-b58c-4b4e-8d9c-e0dce8255295 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:03.103817 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91b6f14-b58c-4b4e-8d9c-e0dce8255295 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:06:03.105277 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91b6f14-b58c-4b4e-8d9c-e0dce8255295 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:06:03.114378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1437/2893] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:06:03 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:06:08.078364 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba921226-5112-4c19-a0da-2151cfd54558 service placement] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:06:08.087866 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba921226-5112-4c19-a0da-2151cfd54558 service placement] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:06:08.091466 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cd302f-7ac6-4879-8d95-327bd2215570 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:06:08.092073 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cd302f-7ac6-4879-8d95-327bd2215570 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:06:08.092356 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cd302f-7ac6-4879-8d95-327bd2215570 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:06:08.092661 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84cd302f-7ac6-4879-8d95-327bd2215570 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:06:08.094485 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-84cd302f-7ac6-4879-8d95-327bd2215570 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:08.178420 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cd302f-7ac6-4879-8d95-327bd2215570 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:06:08.179868 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84cd302f-7ac6-4879-8d95-327bd2215570 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:06:08.198831 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1457/2894] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:06:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:06:11.235185 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91b6f14-b58c-4b4e-8d9c-e0dce8255295 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:06:11.253590 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91b6f14-b58c-4b4e-8d9c-e0dce8255295 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:06:11.264566 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddecb4ca-390f-41f1-aaa2-5d806bcaff8d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:06:11.267824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddecb4ca-390f-41f1-aaa2-5d806bcaff8d service placement] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:06:11.267824 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddecb4ca-390f-41f1-aaa2-5d806bcaff8d service placement] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:06:11.268039 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddecb4ca-390f-41f1-aaa2-5d806bcaff8d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:06:11.273705 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ddecb4ca-390f-41f1-aaa2-5d806bcaff8d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:11.292752 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddecb4ca-390f-41f1-aaa2-5d806bcaff8d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:06:11.294357 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddecb4ca-390f-41f1-aaa2-5d806bcaff8d service placement] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:06:11.312842 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1438/2895] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:06:11 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:06:14.469206 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cd302f-7ac6-4879-8d95-327bd2215570 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:06:14.470680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-84cd302f-7ac6-4879-8d95-327bd2215570 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:14.482802 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84cd302f-7ac6-4879-8d95-327bd2215570 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:06:14.487407 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:06:14.488055 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:06:14.488334 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:06:14.488624 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] PATH_INFO: `/v3/users/ead3628943944bbeabaa2e6a80b99f74` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:06:14.490032 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ead3628943944bbeabaa2e6a80b99f74)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:06:14.492106 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:06:14.538433 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ead3628943944bbeabaa2e6a80b99f74'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:14.539134 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ead3628943944bbeabaa2e6a80b99f74'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:14.539789 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ead3628943944bbeabaa2e6a80b99f74'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:14.551931 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ead3628943944bbeabaa2e6a80b99f74'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:14.558861 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ead3628943944bbeabaa2e6a80b99f74'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:14.571771 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1458/2896] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:06:14 2022] DELETE /identity/v3/users/ead3628943944bbeabaa2e6a80b99f74 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:06:14.749017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddecb4ca-390f-41f1-aaa2-5d806bcaff8d service placement] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:06:14.760487 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddecb4ca-390f-41f1-aaa2-5d806bcaff8d service placement] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:06:14.817663 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:06:14.818983 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:06:14.819266 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:06:14.819541 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] PATH_INFO: `/v3/projects/b586a0d507ca497589a504f9e2d44574` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:06:14.820724 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b586a0d507ca497589a504f9e2d44574)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:06:14.822638 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:06:14.866867 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b586a0d507ca497589a504f9e2d44574', 'request_id': 'req-2e663f49-a037-4cf4-a5a3-829ddce4f17d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:14.873172 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b586a0d507ca497589a504f9e2d44574', 'request_id': 'req-2e663f49-a037-4cf4-a5a3-829ddce4f17d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:14.878625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b586a0d507ca497589a504f9e2d44574', 'request_id': 'req-2e663f49-a037-4cf4-a5a3-829ddce4f17d'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:14.889984 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] The token cache is being invalidate because project b586a0d507ca497589a504f9e2d44574 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:06:14.890269 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:14.890889 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:14.892361 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1439/2897] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:06:14 2022] DELETE /identity/v3/projects/b586a0d507ca497589a504f9e2d44574 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:06:14.896453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:06:14.897937 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:14.910840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d508d2-dfc8-4d3b-8430-f083f0981444 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:06:14.944858 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:06:14.968804 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:06:14.969453 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:06:14.969740 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:06:14.970020 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] PATH_INFO: `/v3/users/3debda601db84e08a5cf0484f510bdb0` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:06:14.972050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3debda601db84e08a5cf0484f510bdb0)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:06:14.974164 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:06:15.020563 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3debda601db84e08a5cf0484f510bdb0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:15.021679 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3debda601db84e08a5cf0484f510bdb0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:15.022319 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3debda601db84e08a5cf0484f510bdb0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:15.034046 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3debda601db84e08a5cf0484f510bdb0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:15.041208 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3debda601db84e08a5cf0484f510bdb0'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:15.054574 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1459/2898] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:06:14 2022] DELETE /identity/v3/users/3debda601db84e08a5cf0484f510bdb0 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:06:15.232563 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:06:15.234028 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:15.243174 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:06:15.257262 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e663f49-a037-4cf4-a5a3-829ddce4f17d admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:06:15.314202 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:06:15.314761 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:06:15.315045 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:06:15.315328 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] PATH_INFO: `/v3/projects/3cb92a2970e6455bb09be55ef55c19e6` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:06:15.316478 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3cb92a2970e6455bb09be55ef55c19e6)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:06:15.318625 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:06:15.368797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3cb92a2970e6455bb09be55ef55c19e6', 'request_id': 'req-15297eba-52c2-4a3f-8b66-02a6510e2084'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:15.375963 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3cb92a2970e6455bb09be55ef55c19e6', 'request_id': 'req-15297eba-52c2-4a3f-8b66-02a6510e2084'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:15.381950 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3cb92a2970e6455bb09be55ef55c19e6', 'request_id': 'req-15297eba-52c2-4a3f-8b66-02a6510e2084'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:15.393017 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] The token cache is being invalidate because project 3cb92a2970e6455bb09be55ef55c19e6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92189) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:06:15.393311 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:15.393961 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-15297eba-52c2-4a3f-8b66-02a6510e2084 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:06:15.395457 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1440/2899] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:06:15 2022] DELETE /identity/v3/projects/3cb92a2970e6455bb09be55ef55c19e6 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:06:25.050288 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=36,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:06:25.390085 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=50,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:06:33.700044 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:06:33.701597 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:33.796216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c8188e-894f-41ac-be51-907abc77e462 admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:06:33.856894 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967d6eb8-7c5e-42fd-97e5-5082ea78a049 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'de49172e2a054096b319a7e4349ecaea', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:06:33.857519 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967d6eb8-7c5e-42fd-97e5-5082ea78a049 service cinder] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:06:33.857811 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967d6eb8-7c5e-42fd-97e5-5082ea78a049 service cinder] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:06:33.858151 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967d6eb8-7c5e-42fd-97e5-5082ea78a049 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:06:33.861096 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-967d6eb8-7c5e-42fd-97e5-5082ea78a049 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:33.926486 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967d6eb8-7c5e-42fd-97e5-5082ea78a049 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:06:33.928528 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967d6eb8-7c5e-42fd-97e5-5082ea78a049 service cinder] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:06:33.937718 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1460/2900] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:06:33 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:06:43.933593 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=77 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:06:44.935452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7fabcc-c77a-4d9a-aa6f-2136031f90cc None None] REQUEST_METHOD: `POST` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:06:44.935956 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7fabcc-c77a-4d9a-aa6f-2136031f90cc None None] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:06:44.936062 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7fabcc-c77a-4d9a-aa6f-2136031f90cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:06:44.954399 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.auth.core [None req-5e7fabcc-c77a-4d9a-aa6f-2136031f90cc None None] MFA Rules not processed for user `6fd1b82319424fb8bd7e50bc0cb00037`. Rule list: `[]` (Enabled: `True`). {{(pid=92189) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 29 05:06:44.955584 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5e7fabcc-c77a-4d9a-aa6f-2136031f90cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:44.963040 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1441/2901] 10.222.0.40 () {58 vars in 958 bytes} [Fri Jul 29 05:06:44 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 29 05:06:45.283702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967d6eb8-7c5e-42fd-97e5-5082ea78a049 service cinder] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:06:45.285197 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-967d6eb8-7c5e-42fd-97e5-5082ea78a049 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:45.352050 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967d6eb8-7c5e-42fd-97e5-5082ea78a049 service cinder] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:06:45.355682 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d69f6c-3b8b-4ee1-83c5-f082758b0e9b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:06:45.356299 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d69f6c-3b8b-4ee1-83c5-f082758b0e9b service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:06:45.356589 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d69f6c-3b8b-4ee1-83c5-f082758b0e9b service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:06:45.356909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d69f6c-3b8b-4ee1-83c5-f082758b0e9b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:06:45.358677 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-45d69f6c-3b8b-4ee1-83c5-f082758b0e9b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:06:45.368296 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d69f6c-3b8b-4ee1-83c5-f082758b0e9b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:06:45.369746 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d69f6c-3b8b-4ee1-83c5-f082758b0e9b service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:06:45.377664 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1461/2902] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:06:45 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:06:54.949787 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=3 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:06:55.374074 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=31 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:07:03.968871 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7fabcc-c77a-4d9a-aa6f-2136031f90cc None None] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:07:03.970424 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-5e7fabcc-c77a-4d9a-aa6f-2136031f90cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:07:04.081378 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7fabcc-c77a-4d9a-aa6f-2136031f90cc None None] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:07:04.087523 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c13b44f-fb49-458e-a802-0fac2e4daf91 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '58c33a72a871432b88da4f0bf1bc514c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:07:04.088148 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13b44f-fb49-458e-a802-0fac2e4daf91 service nova] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:07:04.088436 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13b44f-fb49-458e-a802-0fac2e4daf91 service nova] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:07:04.088757 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c13b44f-fb49-458e-a802-0fac2e4daf91 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:07:04.090533 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-1c13b44f-fb49-458e-a802-0fac2e4daf91 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:07:04.103838 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c13b44f-fb49-458e-a802-0fac2e4daf91 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:07:04.105490 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c13b44f-fb49-458e-a802-0fac2e4daf91 service nova] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:07:04.116801 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1442/2903] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:07:03 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 29 05:07:14.112416 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=46 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:08:39.866720 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d69f6c-3b8b-4ee1-83c5-f082758b0e9b service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:08:39.875544 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d69f6c-3b8b-4ee1-83c5-f082758b0e9b service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:08:39.879287 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482d1de7-57e7-4a7e-a90d-f268b6a931f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:08:39.879896 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d1de7-57e7-4a7e-a90d-f268b6a931f4 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:08:39.880183 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d1de7-57e7-4a7e-a90d-f268b6a931f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:08:39.880465 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d1de7-57e7-4a7e-a90d-f268b6a931f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:08:39.882840 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-482d1de7-57e7-4a7e-a90d-f268b6a931f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:08:39.892225 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482d1de7-57e7-4a7e-a90d-f268b6a931f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:08:39.893711 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482d1de7-57e7-4a7e-a90d-f268b6a931f4 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:08:39.902570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1462/2904] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:08:39 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:08:49.898807 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:08:56.378588 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c13b44f-fb49-458e-a802-0fac2e4daf91 service nova] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:08:56.388948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c13b44f-fb49-458e-a802-0fac2e4daf91 service nova] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:08:56.392698 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7877526-6bb5-4df4-8122-ae5ceb057506 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:08:56.393389 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7877526-6bb5-4df4-8122-ae5ceb057506 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:08:56.393765 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7877526-6bb5-4df4-8122-ae5ceb057506 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:08:56.394104 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7877526-6bb5-4df4-8122-ae5ceb057506 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:08:56.395948 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-a7877526-6bb5-4df4-8122-ae5ceb057506 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:08:56.410118 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7877526-6bb5-4df4-8122-ae5ceb057506 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:08:56.411641 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7877526-6bb5-4df4-8122-ae5ceb057506 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:08:56.423454 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1443/2905] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:08:56 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:08:57.622779 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482d1de7-57e7-4a7e-a90d-f268b6a931f4 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:08:57.624196 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-482d1de7-57e7-4a7e-a90d-f268b6a931f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:08:57.635207 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482d1de7-57e7-4a7e-a90d-f268b6a931f4 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:08:57.638823 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021fa93b-b12e-4293-b216-8a28a9d37c58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:08:57.639425 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021fa93b-b12e-4293-b216-8a28a9d37c58 service neutron] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:08:57.639715 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021fa93b-b12e-4293-b216-8a28a9d37c58 service neutron] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:08:57.639996 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021fa93b-b12e-4293-b216-8a28a9d37c58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:08:57.641702 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-021fa93b-b12e-4293-b216-8a28a9d37c58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:08:57.655831 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021fa93b-b12e-4293-b216-8a28a9d37c58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:08:57.657316 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021fa93b-b12e-4293-b216-8a28a9d37c58 service neutron] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:08:57.669680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1463/2906] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:08:57 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:09:06.418892 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=6 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:09:07.663876 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=6 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:09:09.684086 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7877526-6bb5-4df4-8122-ae5ceb057506 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:09:09.693823 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7877526-6bb5-4df4-8122-ae5ceb057506 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:09:09.697740 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8e80c9-268a-408f-92c1-910bfa1394a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6726ef0c4184040a18a76898bb738d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:09:09.698380 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e80c9-268a-408f-92c1-910bfa1394a0 service neutron] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:09:09.698696 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e80c9-268a-408f-92c1-910bfa1394a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:09:09.699010 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8e80c9-268a-408f-92c1-910bfa1394a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:09:09.700796 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-cd8e80c9-268a-408f-92c1-910bfa1394a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:09:09.764434 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8e80c9-268a-408f-92c1-910bfa1394a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:09:09.765901 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8e80c9-268a-408f-92c1-910bfa1394a0 service neutron] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:09:09.775452 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1444/2907] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:09:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:09:15.922879 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021fa93b-b12e-4293-b216-8a28a9d37c58 service neutron] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:09:15.924570 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-021fa93b-b12e-4293-b216-8a28a9d37c58 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:09:16.017599 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8e80c9-268a-408f-92c1-910bfa1394a0 service neutron] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:09:16.019065 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-cd8e80c9-268a-408f-92c1-910bfa1394a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:09:16.027913 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021fa93b-b12e-4293-b216-8a28a9d37c58 service neutron] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:09:16.032632 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8e80c9-268a-408f-92c1-910bfa1394a0 service neutron] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:09:16.033680 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf72721-cdc7-4b9a-b0e4-337287848ad2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:09:16.034333 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf72721-cdc7-4b9a-b0e4-337287848ad2 service placement] REQUEST_METHOD: `GET` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:09:16.034625 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf72721-cdc7-4b9a-b0e4-337287848ad2 service placement] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:09:16.034909 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf72721-cdc7-4b9a-b0e4-337287848ad2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:09:16.037513 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-fbf72721-cdc7-4b9a-b0e4-337287848ad2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:09:16.037967 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:09:16.038596 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:09:16.038880 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:09:16.039156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] PATH_INFO: `/v3/users/6fd1b82319424fb8bd7e50bc0cb00037` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:09:16.040301 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] RBAC: Authorizing `identity:delete_user(user_id=6fd1b82319424fb8bd7e50bc0cb00037)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:09:16.042314 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:09:16.051189 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf72721-cdc7-4b9a-b0e4-337287848ad2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:09:16.053510 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf72721-cdc7-4b9a-b0e4-337287848ad2 service placement] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:09:16.063741 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1464/2908] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:09:15 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:09:16.092804 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6fd1b82319424fb8bd7e50bc0cb00037'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.101283 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6fd1b82319424fb8bd7e50bc0cb00037'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.101914 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6fd1b82319424fb8bd7e50bc0cb00037'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.102532 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6fd1b82319424fb8bd7e50bc0cb00037'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.109604 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6fd1b82319424fb8bd7e50bc0cb00037'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.126715 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1445/2909] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:09:16 2022] DELETE /identity/v3/users/6fd1b82319424fb8bd7e50bc0cb00037 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 29 05:09:16.307131 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf72721-cdc7-4b9a-b0e4-337287848ad2 service placement] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:09:16.320035 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf72721-cdc7-4b9a-b0e4-337287848ad2 service placement] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:09:16.386277 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:09:16.386923 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:09:16.387216 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:09:16.387505 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] PATH_INFO: `/v3/projects/e10af2e6edcd485caf51326fdc9c17fc` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:09:16.388733 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=e10af2e6edcd485caf51326fdc9c17fc)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:09:16.390639 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:09:16.438636 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e10af2e6edcd485caf51326fdc9c17fc', 'request_id': 'req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.446496 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e10af2e6edcd485caf51326fdc9c17fc', 'request_id': 'req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.454596 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e10af2e6edcd485caf51326fdc9c17fc', 'request_id': 'req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.466098 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] The token cache is being invalidate because project e10af2e6edcd485caf51326fdc9c17fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:09:16.466392 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.467028 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.468554 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1465/2910] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:09:16 2022] DELETE /identity/v3/projects/e10af2e6edcd485caf51326fdc9c17fc => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:09:16.472560 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:09:16.474012 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:09:16.486922 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5ce31c-101a-4289-a6bb-cdba9a02655a admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:09:16.543303 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:09:16.543943 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:09:16.544232 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:09:16.544513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] PATH_INFO: `/v3/users/c58cae9e447143d48edf759a92204e4f` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:09:16.545716 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=c58cae9e447143d48edf759a92204e4f)` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:09:16.548155 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:09:16.553832 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:09:16.599863 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c58cae9e447143d48edf759a92204e4f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.607240 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c58cae9e447143d48edf759a92204e4f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.607912 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c58cae9e447143d48edf759a92204e4f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.608526 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c58cae9e447143d48edf759a92204e4f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.615847 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.notifications [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c58cae9e447143d48edf759a92204e4f'} {{(pid=92189) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.633356 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1446/2911] 10.222.0.40 () {58 vars in 1224 bytes} [Fri Jul 29 05:09:16 2022] DELETE /identity/v3/users/c58cae9e447143d48edf759a92204e4f => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:09:16.802628 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] Authenticating user token {{(pid=92190) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:09:16.804654 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.fernet_utils [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92190) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:09:16.822341 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58af2d6a-9209-4b71-b0a7-85aba1dcb0aa admin admin] Validating token access rules against request {{(pid=92190) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:09:16.841339 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:09:16.884104 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cfc9fe47ed4740a0aaddec545dbaca1a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '25a26196f3ac436c876fd26eb5482e4a', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92190) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:09:16.884793 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] REQUEST_METHOD: `DELETE` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:09:16.885079 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] SCRIPT_NAME: `/identity` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:09:16.885362 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] PATH_INFO: `/v3/projects/b28758e3ebd64fc9864d134a33085838` {{(pid=92190) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:09:16.886542 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] RBAC: Authorizing `identity:delete_project(project_id=b28758e3ebd64fc9864d134a33085838)` {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:09:16.888476 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] RBAC: Authorization granted {{(pid=92190) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:09:16.933458 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b28758e3ebd64fc9864d134a33085838', 'request_id': 'req-4146e09a-9c1b-43c2-84ef-0f9e052c657c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.939447 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b28758e3ebd64fc9864d134a33085838', 'request_id': 'req-4146e09a-9c1b-43c2-84ef-0f9e052c657c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.947317 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b28758e3ebd64fc9864d134a33085838', 'request_id': 'req-4146e09a-9c1b-43c2-84ef-0f9e052c657c'} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.957783 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] The token cache is being invalidate because project b28758e3ebd64fc9864d134a33085838 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=92190) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 29 05:09:16.958105 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.958751 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG keystone.notifications [None req-4146e09a-9c1b-43c2-84ef-0f9e052c657c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=92190) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 29 05:09:16.960349 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: [pid: 92190|app: 0|req: 1466/2912] 10.222.0.40 () {58 vars in 1233 bytes} [Fri Jul 29 05:09:16 2022] DELETE /identity/v3/projects/b28758e3ebd64fc9864d134a33085838 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 29 05:09:26.629156 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:09:26.954850 dv-cinder-851449-2-iscsi devstack@keystone.service[92190]: DEBUG dbcounter [-] [92190] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=92190) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 29 05:10:34.121904 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] Authenticating user token {{(pid=92189) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 29 05:10:34.123405 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:10:34.193512 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188bebcf-9358-4268-8ed9-6df15fd5a5fe admin admin] Validating token access rules against request {{(pid=92189) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 29 05:10:34.253160 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d549cd2-4c25-4e33-add8-045c2fb247dd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e3da5e428049419a9b35494655f52484', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '441bcfe715d44efea69d556677094857', 'project_domain_id': 'default', 'roles': ['admin', 'service', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=92189) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 29 05:10:34.253797 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d549cd2-4c25-4e33-add8-045c2fb247dd service placement] REQUEST_METHOD: `GET` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 29 05:10:34.254087 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d549cd2-4c25-4e33-add8-045c2fb247dd service placement] SCRIPT_NAME: `/identity` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 29 05:10:34.254371 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d549cd2-4c25-4e33-add8-045c2fb247dd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=92189) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 29 05:10:34.256139 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.fernet_utils [None req-4d549cd2-4c25-4e33-add8-045c2fb247dd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=92189) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 29 05:10:34.267383 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d549cd2-4c25-4e33-add8-045c2fb247dd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 29 05:10:34.269611 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d549cd2-4c25-4e33-add8-045c2fb247dd service placement] RBAC: Authorization granted {{(pid=92189) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 29 05:10:34.279513 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: [pid: 92189|app: 0|req: 1447/2913] 10.222.0.40 () {60 vars in 1365 bytes} [Fri Jul 29 05:10:34 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 29 05:10:44.275187 dv-cinder-851449-2-iscsi devstack@keystone.service[92189]: DEBUG dbcounter [-] [92189] Writing DB stats keystone:SELECT=58 {{(pid=92189) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}